Grokking the System Design Interview

0% completed

​

Designing Twitter Search

Twitter users can update their status whenever they like. Each status (called a tweet) consists of plain text and our goal is to design a system that allows searching over all the user tweets.

Try it yourself

Before looking at the solution, try designing it:

2. Requirements and Goals of the System

  • Let's assume Twitter has 1.5 billion total users with 800 million daily active users.
  • On average Twitter gets 400 million tweets every day.
  • The average size of a tweet is 300 bytes.
  • Let's assume there will be 500M searches every day.

.....

.....

.....

Like the course? Get enrolled and start learning!
G

Gary

· 4 years ago

What is stored on each of the index servers when you shard based on tweet object? Is it still a big hash table with key = word, value = set of tweet ids containing that word? Or something else? How does this differ from sharding by word?

Show 1 reply
R

Raj

· 4 years ago

Can you explain how you came up with this math of 5 bytes number to identify the tweetid's uniquely

If we are getting 400M new tweets each day, then how many tweet objects we can expect in five years? 400M * 365 days * 5 years => 730 billion

This means we would need a five bytes number to identify TweetIDs uniquely.

Show 3 replies
S

singeeking

· 3 years ago

In "Designing Twitter Search", I think the article only said how to search by a word, not words (sentences). But the API design has a search param "search_terms" string containing the search terms (maybe some words?).

Is there a lack? Or am I misunderstanding something?

H

Harris

· 4 years ago

can zookeeper replace inddex-builder server?

S

surajwankhade56

· 2 years ago

No mention of latency here. This should be the core problem.

Jonathan Morales

Jonathan Morales

· 2 years ago

Is there anyway the author can update this section and add the functional and non-functional requirements for this system design ?

L L

L L

· 2 years ago

How can we efficiently retrieve a mapping between tweets and the index server? We have to build a reverse index that will map all the TweetID to their index server. [...] We will need to build a Hashtable where the ‘key’ will be the index server number and the ‘value’ will be a HashSet containing all the TweetIDs

First of all, I think there's a mistake in the first part, the mapping is between the index servers and the tweets, not the other way around.

Second, instead of adding yet another mapping (which would also need to be fault tolerant), wouldn't it be simpler to use a Write-Ahead Log (WAL)? Whenever we use an index server to map a given word to a tweet, we can log it to disk in a location dedicated to that index server. If both the leader and the replica of the index se

A

andreachiou88

· a year ago

For the tweets storage, we assumed a server could store 4 TB. For the index, we're assuming a high end server stores a max of 144 GB. Why the difference? Essentially storing text; should be the same/similar?