Grokking the System Design Interview
Vote

0% completed

​

Data Partitioning

Data partitioning is process of dividing a large database (DB) into smaller, more manageable parts called partitions or shards. Each partition is independent and contains a subset of the overall data.

In data partitioning, the dataset is typically partitioned based on a certain criterion, such as data range, data size, or data type. Each partition is then assigned to a separate processing node, which can perform operations on its assigned data subset independently of the others.

.....

.....

.....

Like the course? Get enrolled and start learning!
M

Marvin Xu

· 4 years ago

For round robin you should mention that the order of data coming in determines which partition it lands in. Otherwise people don't know what "i" means.

M

Marvin Xu

· 4 years ago

Also hashed based partitioning is incorrect. If you were to use ID then you could end up lopsided if the IDs are heavily weighted to one partition unless IDs are randomly generated. Instead you should always hash the ID so that it's evenly distributed.

C

chosunone

· 5 months ago

Data partitioning is really a late-stage optimization that locks your schema and access patterns a lot. Reaching for this early is an easy way to make poor choices. I think it should be emphasized that data partitioning should come after you have identified stable access patterns and schema.

Piyush Kuhikar

Piyush Kuhikar

· 19 days ago

Understanding Directory-Based Partitioning

In this explanation, we will break down the concept of directory-based partitioning in a way that’s easy to grasp, including its benefits, drawbacks, and how it fits into the broader context of system design.

What is Partitioning?

Partitioning is the process of dividing a large dataset into smaller, more manageable pieces or partitions. This is often done to improve performance and scalability in systems dealing with vast amounts of data.

Types of Partitioning

  • Range-Based Partitioning: Divides data based on ranges of values.
  • Hash-Based Partitioning: Distributes data using a hash function, spreading it evenly across partitions.
  • Directory-Based Partitioning (Focus of Our Discussion): Uses a directory (lookup table
M

Marvin Xu

· 4 years ago

You should mention that vertical partitioning means partition by column

Show 1 reply
SOHEL RANA

SOHEL RANA

· 3 months ago

It will be helpful is updation history mentioned in articles, so that user can get if any comments or suggestion has been granted. I think blog is modified based on some suggestion but is not mentioned so when i go through comments feels like already acknowleged.

Sanket Doshi

Sanket Doshi

· a month ago

For example, consider an e-commerce website that stores customer data in a database table. The website might partition the customer table vertically based on the type of data, so that personal information such as name and address are stored in one shard, while order history and payment information are stored in another shard. This way, when a customer logs in and their order history needs to be accessed, the query can be directed to the appropriate shard, minimizing the amount of data that needs to be scanned.

This eample for vertical partitioning is wrong.

Show 1 reply