I am trying to create a simple alert app for some friends.
Basically i want to be able to extract data "price" and "stock availability" from a webpage like the folowing two:
What ever you do: Don't use regular expressions to parse HTML or bad things will happen. Use a parser instead.
You are probably best off loading the HTML code into a DOM parser like this one and searching for the "pricing" table. However, any kind of scraping you do can break whenever they change their page layout, and is probably illegal without their consent.
The best way, though, would be to talk to the people who run the site, and see whether they have alternative, more reliable forms of data delivery (Web services, RSS, or database exports come to mind).
Hi Mike,
1st, asking this question goes too into details. 2nd, extracting data from a website might not be legitimate. However, I have hints:
Use Firebug or Chrome/Safari Inspector to explore the HTML content and pattern of interesting information
Test your RegEx to see if the match. You may need do it many times (multi-pass parsing/extraction)
Write a client via cURL or even much simpler, use file_get_contents (NOTE that some hosting disable loading URLs with file_get_contents)
For me, I'd better use Tidy to convert to valid XHTML and then use XPath to extract data, instead of RegEx. Why? Because XHTML is not regular and XPath is very flexible. You can learn XSLT to transform.
Good luck!
As others have pointed out, it probably isn't a good idea to do this for many reasons, but here's the code anyways:
$c = curl_init();
curl_setopt($c, CURLOPT_URL, 'http://www.sparkfun.com/commerce/product_info.php?products_id=9279');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_HEADER, 0);
$d = curl_exec($c);
// Get the value from the first th of table.pricing
preg_match('#<table class="pricing">.*?<th>\$(.*?)</th>#s', $d, $match);
$price = $match[1];
The price of the product can now be found in the $price variable.
Note that you should ask permission first, I'm sure the owner of the site isn't pleased that you using their data without permission and fetching 11kb of data to get just one floating point number.
As I'm not adept in using XML parsers, I'd be interested to see what a 'proper' solution would look like.
$content = file_get_contents('http://www.sparkfun.com/commerce/product_info.php?products_id=9279');
preg_match('#<tr><th>(.*)</th> <td><b>price</b></td></tr>#', $content, $match);
$price = $match[1];
preg_match('#<input type="hidden" name="quantity_on_hand" value="(.*?)">#', $content, $match);
$in_stock = $match[1];
echo "Price: $price - Availability: $in_stock\n";
It's called screen scraping, in case you need to google for it.
I would suggest that you use a dom parser and xpath expressions instead. Feed the HTML through HtmlTidy first, to ensure that it's valid markup.
For example:
$html = file_get_contents("http://www.example.com");
$html = tidy_repair_string($html);
$doc = new DomDocument();
$doc->loadHtml($html);
$xpath = new DomXPath($doc);
// Now query the document:
foreach ($xpath->query('//table[@class="pricing"]/th') as $node) {
echo $node, "\n";
}
i tried and it works fine. but i have another problem. this example extract only the first data that he find. but if in the webpage there is something like that: name 1 ... ...
how can i display all the name?