Hacker News new | past | comments | ask | show | jobs | submit login

If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations.

If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.




It's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), then scaling it back down after the operation because it is very costly and priced by hour. This makes it complicated and expensive on dynamo.

It seems the only storage services compatible with variable unlimited bursts of concurrency are S3 and SimpleDB. S3 comes with many problems for handling structured data (no update of records only replace, locking, listing items is slow/costly, etc.). SimpleDB is no longer being iterated, is limited to 10gb per domain, and looks like it's being slowly phased out.

It seems like massively parallel lambdas depend on few fetches of large blobs of data - which is basically batch-processing EMR-style, or better suited to redshift. Not something that opens the door for novel use-cases.

I would have really liked for dynamodb to be more of a service than a vm. I wish its concurrency was unlimited and you paid for usage rather than time. Basically DynamoDB with SimpleDB pricing.


Just use RDS and S^3 for the blobs. RDS can do tens of thousands of index lookups a second.

If you only need one index, then just name your s3 document by the compound index value and call it a day.

Otherwise, just use RDS for everything.


From the RDS FAQ:

> In order to maximize your workload’s throughput on Amazon Aurora, we recommend building your applications to drive a large number of concurrent queries.

Perrrrfect. Thanks!

https://aws.amazon.com/rds/aurora/faqs/




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: