Designing Twitter

Caching full tweet JSON for timeline is usually not ideal.

A more common design is I believe

Timeline cache

user123→ [tweet1,tweet2,tweet3,...]

So we should Store only tweet IDs.

Tweet cache

tweet1 → tweet data

tweet2 → tweet data

We should store tweet content separately.

Because tweet may appear in 10M timelines.

If stored fully everywhere we are creating huge duplication.

If stored once :

Timeline = IDs

Tweet Cache = actual content

So this gives much smaller memory footprint.

Do we need just one API in this problem. Looks like the section is incomplete

You correctly say that sharding based on tweet id fixes the hot user problem but then increases latency since you need to query all shards (in the case of a shard taking longer to respond, your latency increases). This is fine.

Your second suggestion is epoch time, and that doesn’t work cuz servers holding recent info with be much hotter than other serivera (due to range based partitioning)

What’s not fine is claiming that if you combine epoch time with a sequence number to be your tweet id that automatically all your problems are solved. You go back to using your hashing function and you still need to query all your db servers.

Do correct me, but I don’t see how that fixes the latency problem.

Posting Tweet is a single API with media []? Because uploading media is a separate process should we break this API down into two?

the part at the end which starts with:

"We would need 31 bits to store this number. Since on average we are expecting 1150 new tweets per second, we can allocate 17 bits to store auto incremented sequence; this will make our TweetID 48 bits long. So, every second we can store (2^17 => 130K) new tweets. We can reset our auto incrementing sequence every second. For fault tolerance and better performance, we can have two database servers to generate auto-incrementing keys for us, one generating even numbered keys and the other generating odd numbered keys."

till the end of the solution.. belongs before section 9

How do we know it takes 31 bits for epoch seconds and 17 bits for auto incrementing sequence? Where did these numbers come from?

To handle this, a more intelligent LB solution can be placed that periodically queries backend server about their load and adjusts traffic based on that.

Whats the ideal one here?

Trying to understand data sharding. If data is sharded by tweetid, we can fetch top 10 latest tweets globally, but how would we do for a user A lets say? Would not it still have to find first a pool of top tweets for that particular user which would require alot of more work?

Benefits of putting creation time in the PK are questionable, since timeline is created only from tweets by the users you follow you now need to read vast amounts of data to get any hit. Same with trying to read some other user's page.