How to extract content from other websites automatically?

Question

I want to extract a specific data from the website from its pages...

I dont want to get all the contents of a specific page but i need only some portion (may be data only inside a table or content_div) and i want to do it repeatedly along all the pages of the website..

How can i do that?

Answer 1

Use curl to retreive the content and xPath to select the individual elements.

Be aware of copyright though.

Answer 2

You need the php crawler. The key is to use string manipulatin functions such as strstr, strpos and substr.

Answer 3

There are ways to do this. Just for fun I created a windows app that went through my account on a well know social network, looked into the correct places and logged the information into an xml file. This information would then be imported elsewhere. However, this sort of application can be used for motives I don't agree with so I never uploaded this.

I would recommend using RSS feeds to extract content.

Answer 4

I think, you need to implement something like a spider. You can make an XMLHTTP request and get the content and then do a parsing.

Answer 5

"extracting content from other websites" is called screen scraping or web scraping.

simple html dom parser is the easiest way(I know) of doing it.

Answer 6

How about the imacros tool for firefox to do repetitive tasks? will that be used to get the data from a site which has a standard format of displaying data?