Which database engine should I use for a web crawler, InnoDB or MYiSAM? I have two PC's, each with 1TB hard drives. If one fills up, I'd like for it to save to the other PC automatically, but reads should go to the correct PC; how do I do that?
views:
64answers:
3
+1
A:
As for the first part of your question, it rather depends on you precise implementation. If you are going to have a single crawler limited by network bandwidth, then MYiSAM can be quicker. If you are using multiple crawlers then InnoDB will give you advantages such as transactions which may help.
AFAIK MySQL doesn't support the hardware configuration you are suggesting. If you need large storage you may wan tot look at MySQL Cluster.
Jaydee
2010-08-17 15:29:18
@Jaydee:i am using multithread programming using 2 system to crawl,in that case which is good MyISAM?, how we can do FullText Search in InnoDB?
Jesvin
2010-08-17 16:17:54
A:
This previous question may help you
http://stackoverflow.com/questions/1381186/fulltext-search-with-innodb
Jaydee
2010-08-19 12:16:36
A:
MyISAM is the first choice, because you will have write only operations and crawlers -- even run in parallel -- will be configured -- I suppose -- to crawl different domains/urls. So you do not need to take care of access conflicts.
When writing a lot of data, especially text!, to Mysql avoid transactions, indexes, etc., because it will slow down MySQL drastically.
Skarab
2010-08-19 12:31:16