NNTP usenet download messages

Question

I would like to download usenet messages to my personal Mysql database, so without using any newsreader. At the moment i'm using IpWorks component for .NET . I've created a simple app in Vb.Net, It works, but it's realy slow... to download and to do the INSERT INTO . Anyone know some better method to "massive" download message and insert it into a simple mysql table?

Answer 1

You might want to try using a bulk insert or a delayed insert instead of entering items row by row sequentially.

Additionally, a lot of this can be done asynchronously. Most of the work you are doing (writing to the database and downloading the content from nntp) is not bound to resources on your machine. You can perform the downloading and reading of the nntp content in a thread separate from the ones that perform the inserts (you should buffer the content you want to insert before you send it to mysql).

Basically, you want a producer consumer pattern.