Sharding

1 / Horizontal Data Partitioning

Sharding was the natural extension of Consistent Hashing — once keys were deterministically routed to specific nodes, each node managed its own independent data partition. In Shard and Cairn, sharding meant distributing cached key-value pairs across multiple cache nodes so that no single node held the entire dataset.

2 / Shard Key Selection

The effectiveness of sharding depends entirely on shard key selection. A poorly chosen shard key creates hotspots — one shard receives disproportionate traffic while others are underutilized. In the cache implementations, the key itself (after Murmur3 hashing) served as the shard key, which provided uniform distribution. In relational database contexts, shard key selection is more nuanced — it must balance query locality with distribution uniformity.

3 / Query Aggregation

Multi-node queries — operations that need data from multiple shards — were the most complex aspect. In a distributed cache, a 'get all keys' operation requires querying every shard and aggregating results. This scatter-gather pattern introduces latency proportional to the slowest shard, which is why sharded architectures prefer single-shard operations wherever possible.