I can easily do my work with text files and database both. But i want to know which one is good in terms of response time. If text files is not the best solution for fastest result then which database is best, again in terms of response time. I can trade off size for response time. Please help me.
views:
47answers:
3
+1
A:
Databases are databases for a very good reason, they are HIGHLY optimized for a) Storing your data in an efficient manner and b) Retrieving it in an efficient manner.
You CANNOT perform queries on a text file, so there is no sorting, ordering, optimizing etc. As far as security goes, a text file is not password protected and someone may easily view the contents of the file, this is not the case with a database. The list is endless of course, and you really should be using a database!
If you are looking for a solution SIMILAR to a text file, try SQLite, http://www.sqlite.org/
webfac
2010-08-03 11:12:32
thanks you very much
Kuri
2010-08-03 11:48:33
I perform queries against text files all the time - I use grep, and get much better performance on my particular dataset than I get from the same dataset loaded into a database.This answer is simplistic nonsense. Flat-files are often the right tool for the job. As are relational databases; inverted-file indices; document-oriented datastores; column-oriented key-value stores; and, graph-oriented non-relational databases.You need to understand the shape of your data, and your use cases before you can suggest "you really should be using a database". -1.
Recurse
2010-08-03 11:52:36
@ Recurse - You have a valid point, but don't shoot posts down when they indeed do have valid standpoint! It all depends on the scenario. If this was being performed on a web based server, then NO one would NOT use flat files stored locally as this would a) be a security hole and b) There is little support for indexing in a text file, regardless of what system you choose to use. Remember that the system you use (grep) in your case is an overhead unto it's own. While I completely respect your opinion and DO agree that flat files CAN be a viable 0ption, they more than likely are not.
webfac
2010-08-03 12:02:41
A:
Beside what was already mentioned, databases provide concurrent access, error recovery, distribution over a network (client/server) and transactions.
The performance gains with databases depend on indexes and query optimization, so in some cases, when you don't need any of the above, and a certain query needs to sequentially scan all data anyway, using a text file might be faster. But in general, with most serious applications, you will find yourself very soon be implementing some of the functionality of a database yourself, and in that case, you'd rather use a real database in the first place.
thieger
2010-08-03 11:34:44
+1
A:
The index overhead of a database means that a CSV file or similar will be more space efficient. Assuming the database is using a B+Tree or similar for its data file (ie, MySQL using InnoDB), the CSV file will (assuming the use of a decent extent-based fs) also be more efficient at sequential processing.
However
If you need to retrieve a specific record then a database or datastore is specifically designed for efficient random access. Moreover if you are wanting to perform more complicated queries than a simple lookup, a database has numerous features available to provide consistency and integrity. If you are needing shared read/write access then you will find a database/datastore invaluable.
Still
If you are just looking for a simple way to persist data you will be using in-memory; you don't require concurrent access; or, your access pattern is record-oriented sequential batch-processing, then you really can't go past the simplicity, ease-of-development/debugging, and highly optimised OS-mediated performance of the humble CSV file.
Anyone who tells you not to bother thinking about your specific use patterns and requirement and "Just use a database", is simply wrong.
Recurse
2010-08-03 11:47:36
@Kuri - This is a good point as well, and I stress the importance of indexing and relationships. You can forget about one to many relationships being faster in flat files that databases, and cross referencing records may become cluttered. I personally believe the database is designed to manage data more effectively. And using MySQL's MyISAM will result in much faster results when using complex queries.
webfac
2010-08-03 12:05:46
@kuri pay attention to what @webfac says about cross-referencing and indexing. As soon as you start wanting to do any sort of correlative query you are much better off with a database. As soon as you need random-access, either a database or a datastore/filesystem will be much better than a flatfile. Sometimes the added complexity isn't worth it, but that is uncommon unless it is a trivial task.
Recurse
2010-08-03 12:20:48