Grokking the System Design Interview
Vote

0% completed

​

Data Partitioning

One machine can only hold so much data, and it can only handle so many writes. When a table grows past what one machine can serve, you split it across several machines.

Data partitioning is the act of splitting data into parts and putting each part on a different machine. Each part is called a partition or a shard.

The words are used loosely in practice. Partitioning is the general idea. Sharding usually means partitioning across separate machines.

Horizontal and Vertical Partitioning

There are two ways to cut a table, and they solve different problems.

**Horizontal partition

.....

.....

.....

Like the course? Get enrolled and start learning!
M

Marvin Xu

· 4 years ago

For round robin you should mention that the order of data coming in determines which partition it lands in. Otherwise people don't know what "i" means.

M

Marvin Xu

· 4 years ago

Also hashed based partitioning is incorrect. If you were to use ID then you could end up lopsided if the IDs are heavily weighted to one partition unless IDs are randomly generated. Instead you should always hash the ID so that it's evenly distributed.

C

chosunone

· 5 months ago

Data partitioning is really a late-stage optimization that locks your schema and access patterns a lot. Reaching for this early is an easy way to make poor choices. I think it should be emphasized that data partitioning should come after you have identified stable access patterns and schema.

Piyush Kuhikar

Piyush Kuhikar

· a month ago

Understanding Directory-Based Partitioning

In this explanation, we will break down the concept of directory-based partitioning in a way that’s easy to grasp, including its benefits, drawbacks, and how it fits into the broader context of system design.

What is Partitioning?

Partitioning is the process of dividing a large dataset into smaller, more manageable pieces or partitions. This is often done to improve performance and scalability in systems dealing with vast amounts of data.

Types of Partitioning

  • Range-Based Partitioning: Divides data based on ranges of values.
  • Hash-Based Partitioning: Distributes data using a hash function, spreading it evenly across partitions.
  • Directory-Based Partitioning (Focus of Our Discussion): Uses a directory (lookup table
M

Marvin Xu

· 4 years ago

You should mention that vertical partitioning means partition by column

Show 1 reply
SOHEL RANA

SOHEL RANA

· 3 months ago

It will be helpful is updation history mentioned in articles, so that user can get if any comments or suggestion has been granted. I think blog is modified based on some suggestion but is not mentioned so when i go through comments feels like already acknowleged.

Sanket Doshi

Sanket Doshi

· 2 months ago

For example, consider an e-commerce website that stores customer data in a database table. The website might partition the customer table vertically based on the type of data, so that personal information such as name and address are stored in one shard, while order history and payment information are stored in another shard. This way, when a customer logs in and their order history needs to be accessed, the query can be directed to the appropriate shard, minimizing the amount of data that needs to be scanned.

This eample for vertical partitioning is wrong.

Show 1 reply