There are many limitations in dynamo db despite it being highly scalable.<ol><li>Item size limit<ol style="list-style-type:decimal;"><li>Each record can only have a maximum of 400KB. So it is best not to store any large data in it. It is best to use alternative approaches</li></ol></li><li>Page size limit<ol style="list-style-type:decimal;"><li>One request can return only 1MB worth of records</li></ol></li><li>Partition limit<ol style="list-style-type:decimal;"><li>Can have many partitions. Each partition can have 3,000 Read Capacity Units (RCUs) and up to 1,000 Write Capacity Units (WCUs) per second.</li></ol></li></ol><ol style="list-style-type:decimal;"><li>ConditionExpression, ProjectionExpression, UpdateExpression &amp; FilterExpression length - up to 4KB</li><li>GSIs (Global Secondary Indexes) that use a different attribute as partition key, are limited to 20 per table.</li></ol>Things to keep in mind while designing data model with DynamoDB<ol style="list-style-type:decimal;"><li>Store objects in&nbsp; Amazon S3 and reference in DynamoDB: Instead of storing the large object directly in DynamoDB, you can store the object in Amazon S3 (Simple Storage Service) and store a reference to the object in DynamoDB. The reference could be an S3 object key or URL. This allows you to leverage the scalability and cost-effectiveness of S3 for large data storage, while still using DynamoDB for efficient querying and indexing of the references. &nbsp;</li></ol><figure class="image"><img src="https://s3.ap-south-1.amazonaws.com/cdn.nirmitee.io/dynamo_storing_large_objects_4de4d6459e.png"></figure><ol style="list-style-type:decimal;"><li>Optimise your data model: Review your data model to ensure efficient access patterns and avoid hot partitions. Hot partitions occur when a small subset of keys or items receives a disproportionately high number of read requests, causing capacity issues. By designing your data model with a well-distributed access pattern, you can prevent hot partitions and balance the read workload across multiple partitions.<ol style="list-style-type:lower-latin;"><li>Split objects into smaller pieces: If your larger objects can be logically divided into smaller pieces or chunks, you can split them into manageable parts and store each part as a separate item in DynamoDB. You can use a unique identifier or a composite key to associate the related parts of the object. When retrieving the object, you can fetch and combine the individual parts to reconstruct the complete object.</li><li>Implement a hierarchical structure: If your larger objects have a hierarchical or nested structure, you can break them down into smaller components and store them as separate items in DynamoDB. Each item represents a specific part of the object and contains references to related parts or sub-objects. This allows you to manage and retrieve the object's components individually while maintaining the relationship between them</li><li>Understand your access patterns: Gain a thorough understanding of the read and write patterns of your application. This includes identifying the types of queries you need to perform, the expected data retrieval patterns, and the frequency of data updates. This understanding will help you design an efficient data model that aligns with your application's requirements.</li></ol></li><li>Distribute data across partitions: Consider implementing a data partitioning strategy to distribute the data across multiple partitions. This involves choosing an appropriate partition key that evenly distributes the workload across partitions. By spreading the data across multiple partitions, you can effectively utilise the provisioned capacity for each partition and avoid overloading a single partition. The choice of partition key can significantly affect performance and scalability. It should be based on thee access patterns and data distribution characteristics of your application.</li><li><img src="https://s3.ap-south-1.amazonaws.com/cdn.nirmitee.io/dynamodb_partitions_c2d497aa8f.png"></li></ol><ol><li>Use paginated queries:&nbsp;When you perform a query or scan operation in DynamoDB, the response is returned in pages of data. Each page contains a subset of the total result set. By default, DynamoDB returns a maximum of 1 MB of data per page.<ol style="list-style-type:decimal;"><li>If a query or scan operation results in more than 1 MB of data, you can use pagination techniques to retrieve the remaining data. DynamoDB provides a `LastEvaluatedKey` parameter in the response, which you can use as a starting point for the next page of results. You can iterate through the pages by repeatedly making requests and providing the `LastEvaluatedKey` from the previous response.</li></ol></li></ol><ol style="list-style-type:decimal;"><li>If an expression exceeds the limit of 4KB, you may need to modify or optimise the expression, consider splitting it into multiple expression</li><li>Using sort keys to organise data: If you're using a composite primary key (partition key + sort key), consider how the sort key will be used for querying and sorting data. It should support the specific query patterns you need and allow for efficient sorting and filtering.</li><li>Utilise secondary indexes effectively: DynamoDB allows you to create secondary indexes to enable additional query patterns. Consider the specific queries you need to perform and create secondary indexes accordingly. However, be mindful of the impact on performance and cost, as secondary indexes consume additional storage and throughput.</li><li>Plan for read and write capacity: Provision read and write capacity based on the expected workload of your application. Monitor and adjust the provisioned capacity as needed to ensure optimal performance. Utilise DynamoDB Auto Scaling to automatically adjust capacity based on actual usage patterns</li><li>Last but not least, It's also important to balance performance, scalability, and cost-effectiveness while ensuring that the model aligns with your desired access patterns.</li></ol>&nbsp;References:<ol style="list-style-type:decimal;"><li><a href="https://aws.amazon.com/blogs/database/large-object-storage-strategies-for-amazon-dynamodb/">https://aws.amazon.com/blogs/database/large-object-storage-strategies-for-amazon-dynamodb/</a></li><li><a href="https://www.alexdebrie.com/posts/dynamodb-partitions/">https://www.alexdebrie.com/posts/dynamodb-partitions/</a></li><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.html">https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.html</a></li></ol>

large_Designer (8).png

Designing data model with DynamoDB

Similar stories

Get our stories delivered

© Nirmitee.io. 2023, India. All rights reserved