tags:

views:

38

answers:

1
+1  Q: 

Bulk download of web pages using Qt

I want to write a program using Qt that downloads a lot of HTML web pages, about 5000, from one site every day. After downloading that pages I need to extract some data using DOM Query, using the WebKit module, and then store that data in a database.

Which is the best/correct/efficient way to do that, in particular the download and analysis phase? How do I handle that amount of requests and how to create the "download manager"?

+2  A: 

To download the pages it makes sense to use a dedicated library like libcurl

PiedPiper  2010-10-11 22:49:29

related questions

Autosizing Textarea
Regular expression for parsing links from a webpage?
What are good tools for creating compiled HTML help files (.chm)?
Looking for WYSIWYG HTML editor
Any reason not to start using the HTML 5 doctype?
HTML comments break down
HTML Comments Markup
Setting a div's height in HTML with CSS
Wrapping lists into columns
Is a "Confirm Email" input good practice when user changes email address?
<XMP> Tag
HTML version choice
Options for HTML scraping?
How do you disable browser Autocomplete on web form field / input tag?
How do I make a checkbox toggle from clicking on the text label as well?
Html CSS Editor
Wordpress theme development offline tools
How do I give my web sites an icon for iPhone?
In HTML, how to word-break on a dash?
Detecting font in JavaScript
How do you test layout design across multiple browsers/OSs?
How do I print an HTML document from a web service?
Multiple submit buttons on a HTML form
How can I determine a web user's time zone?
Why doesn't the percentage width child in absolutely positioned parent work in IE7?