Guidance required: FIrst time gonna work with real high end database (size = 50GB).

Question

I got a project of designing a Database. This is going to be my first big scale project. Good thing about it is information is mostly organized & currently stored in text files. The size of this information is 50GB. There are going to be few millions of records in each Table. Its going to have around 50 tables. I need to provide a web interface for searching & browsing. I'm going to use MySQL DBMS.

I've never worked with a database more than 200MB before. So, speed & performance was never a concern but I followed things like normalization & Indexes. I never used any kind of testing/benchmarking/queryOptimization/whatever because I never had to care about them.

But here the purpose of creating a database is to make it quickly searchable. So, I need to consider all possible aspects in design.

I was browsing archives & found:

http://stackoverflow.com/questions/1981526/what-should-every-developer-know-about-databases

http://stackoverflow.com/questions/621884/database-development-mistakes-made-by-app-developers

I'm gonna keep the points mentioned in above answers in mind.

What else should I know? What else should I keep in mind?

Answer 1

Just index the columns you'll be searching on and you'll be fine. There is nothing in particular that you should know besides that.

Remember that worrying so early about optimizations is probably not a good idea. Write the software and only if it goes too slow think about making it go faster.

Answer 2

Also take a look at

Steve Huffman on Lessons Learned at Reddit