I keep seeing people get confused on DynamoDB having 10GB partitions and thinking they need to design around it. This happened twice in the last week!
Writing a long tweet to clarify.
Two main takeaways:
1. The '10GB partition size' is rarely visible to you as a user.
2. You are not limited to 10GB of data for a given partition key (**if you don't have an LSI on your table**).
Explanation:
First, some vocab:
The term 'partition' refers to a co-located subset of your data within a DynamoDB table. Partitions are ~10GB in size but can be larger or smaller. DynamoDB is a multi-tenant system with a huge fleet of storage nodes, and each storage node contains many partitions from many different tables.
The term 'item collection' is used to refer to the set of records that share the same partition key in your table. Within the item collection, items are sorted according to the sort key.
-----
DynamoDB assigns an item to a partition using the partition key (and, occasionally, the sort key).
In most cases, a single partition will contain multiple item collections. Also, a single item collection will often be contained on the same partition. But this is not always the case!
If your item collection exceeds 10GB, DynamoDB will split it across multiple partitions.
If you are driving heavy traffic to your item collection and DynamoDB thinks it could serve it better across multiple partitions, it might split it across multiple partitions.
But there's not necessarily a 1:1 mapping from partitions to item collections nor from item collections to partitions.
However, if your table has a local secondary index (LSI), the rules change. Now, your item collection *must* fit within a single partition. This means the item collection any LSI replicas of it must be under 10GB.
This is the only time the 10GB limit applies. This is pretty rare -- I recommend against LSIs in almost all scenarios, partly because of this limitation.
But if you don't have an LSI, you don't need to worry about keeping your item collections under 10GB (for capacity reasons -- there may be query-based reasons to keep it smaller)!
✨ The more you know