Can mysql handle a dataset of 50gb (only text) efficiently ? If not, what database technologies should I use ?
thanks
+6
A:
Technically, I would say yes. MySQL can handle 50GB of data, efficiently.
If you are looking for a few examples, Facebook moved to Cassandra only after it was storing over 7 Terabytes of inbox data.
Wikipedia also handles hundreds of Gigabytes of text data in MySQL.
Daniel Vassallo
2010-03-06 08:46:42
If you're going to cite FB + inbox data, you should finish by recalling that the reason they moved to Cassandra was that doing this in MySQL was so painful. :)
jbellis
2010-03-06 15:20:54
@jbellis: True. My point was that they migrated to Cassandra much later than 50Gb. I modified my answer to make this a bit more clear The source I cited explains the details.
Daniel Vassallo
2010-03-07 09:01:13
+3
A:
Any backend that uses b-trees (like all the popular ones for MySQL) get dramatically slower when the index doesn't fit in RAM anymore. Depending on your query needs, Cassandra might be a good fit, or Lucandra (Lucene + Cassandra) -- http://blog.sematext.com/2010/02/09/lucandra-a-cassandra-based-lucene-backend/
jbellis
2010-03-06 15:23:22