Is there a way to crawl all facebook fan pages and collect some information? like for example crawling facebook fan pages and save their names, or how many fans, etc? Or at least, do you have a hint of how this could be possibly done?
how to make it only crawl facebook fan pages?
2010-04-02 05:30:19
Start by choosing a language, python is a good choice for crawlers. Then spot a pattern on the fan pages urls so your web crawler starts to pick them up recursively and access them. Then using regular expressions get the data you need.Sounds hard but it's not.
Ben
2010-04-02 05:34:32
the problem is: there is no pattern for fan pages urls, you need to figure them out through looking at the content itself which is why i asked the question
2010-04-02 05:41:03
sure there is, there always is, maybe not in the url itself but in the container of the urls, the id of the a tag, the class, etc. Remember that you'll get html, that's what you'll be parsing.
Ben
2010-04-02 05:49:56
+1
A:
Write a crawler.
I used Coca-Cola's page as an experiment: http://www.facebook.com/cocacola?v=wall
Parse out the "Fans" div, which contains an "All Fans" link. View source in your web browser, it looks like this: /social_graph.php?node_id=40796308305&class=FanManager
Turn that into a facebook URL and crawl it: http://www.facebook.com/social_graph.php?node_id=40796308305&class=FanManager
Parse out the fans, then parse out the "Next page" link.
Repeat, ad nauseum.
Throttle your requests so facebook doesn't blacklist you.
Stephen
2010-04-02 05:55:08
+1
A:
First select a page that contains your desired category for pages:
For Example: http://www.facebook.com/pages/ or http://www.facebook.com/pages/?browse&ps=93
Then use a crawler to get all pages links.
Now you can parse each page separately using extracted links.
You can use simple html dom for crawling.
NAVEED
2010-04-02 05:57:02
A:
RE: Stephen:
Could you please explain your response in a little more detail? I'm not the most fluent programmer, but would like to be able to export the links to the profiles of all of the fans of a band into a csv file. Could you explain how you parse out the "fans" divider and view its source? In addition - how do you parse out the fans, then parse out the next page link?