Get info from multiple files, match it and then display to end user, what is fastest?

Question

Hi, I need to build a website where we display data that is refreshed every 5minutes in a text file with a | separator.

I currently use Java to do this.

What I do now: I grab the textfile for every request through the website and process it and then display the data to the end user, this works fine, since Java can go through like 5000 lines of data fast, and when I filter it it is still extremely fast.

However now the management wants the following: They added 3 textfiles with the | separator to it, and now want me to also read those files and match the information on certain fields, and if there is a match also display that information to the end user.

I think soon enough, although Java is fast, I will run into trouble when 10 people want that information and I have to run through 4 total files matching the information.

What can I do to make this process super fast?

My Creative solutions so far:

-Leave it this way, since Java is fast and end users can wait (probably less then 1second)

-Have a background process that dumps new data into a MySQL database every 5minutes, since databases are extremely good at getting the same data from multiple tables.

Thank you!

Answer 1

Both solutions are good. A few things to keep in mind:

• If you're going to keep the user waiting for more than 1 second, consider making a landing page and showing a "loading info - please wait" message while your page loads the information (using AJAX, for example).
• Chances are management will add more files later. Coding a database solution now might save you from hacking it now and having to do it the right way later anyway.
• Other reasons to use the database are the DBMS extra advantages in caching, locking, joining...

Answer 2

I'm not really a Java guy, but I'm pretty sure you should be able to read these files into cache for 5 minutes... this way the data is combined and stored in memory... In addition to putting it in memory, you could store it in a hash table or some other data structure that makes searching faster (find value versus evaluate each value)...