When your application hits massive scale, vertical scaling (buying a bigger server) eventually reaches a point of diminishing returns. To scale your database to handle billions of rows, you need to shard.
What is Sharding?
Sharding is the process of splitting a single logical dataset into multiple physical databases, known as shards. Each shard contains a subset of the total data, allowing you to distribute the load across many servers.
Horizontal vs Vertical Partitioning
Vertical partitioning involves splitting a table by columns (e.g., moving rare columns to a separate table). Horizontal partitioning—which is what sharding is—involves splitting a table by rows. For example, users with IDs 1-1000 go to Shard A, while 1001-2000 go to Shard B.
The Sharding Key: Choosing the right sharding key is the most critical decision. A poor key leads to "hot spots" (where one shard does all the work) and makes joins across shards incredibly expensive.
The Trade-offs
- Complexity: Your application must now know which shard to query.
- Joins: Joining data across shards is difficult and usually avoided by denormalizing data.
- Transactions: Maintaining ACID properties across shards requires distributed transaction managers, which add significant latency.