How to get all Twitter links?

views:

372

answers:

+4 Q:

How to get all Twitter links?

There are lots of sites offering additional services to Twitter:

hashtags.org
tweetmeme.com
repeets.com
dailyrt.com
backtweets.com

They all have something in common: They seem to get all updates from Twitter. Otherwise, they couldn't offer such complex services with high quality results.

But how do they do this? Twitter has a XMPP feed (Firehose) but not everybody is allowed to get data from it.

Are there any approaches you can recommend or which you think they are using? The normal API is limited too much, isn't it?

+5 A:

Quoted from http://apiwiki.twitter.com/Rate-limiting

Whitelisting

Some applications find that the default limit proves insufficient. Under such circumstances, we offer whitelisting. It is possible to whitelist both accounts and IP addresses. Each whitelisted entity, whether an account or IP address, is allowed 20000 requests per hour. If you are developing an application that should be considered for whitelisting, please fill out the whitelisting request form. Our manual review process can take up to a week. If you have a whitelisting that needs to be updated through the addition or removal of IP addresses, reapply with an explanation of the change. Approval or rejection for whitelisting requests is emailed to the email address associated with the account that filed the application.

Amber 2009-08-02 23:19:32

But even if you're whitelisted there are still the main problems: The API results are cached and not all tweets can be displayed, e.g. if you search for "and". This word is used so often that not all tweets can be displayed in the possible 1500 updates.

2009-08-02 23:53:15

To be fair, you don't actually need **all** the tweets, just a majority of them. Aim for ``good enough'' rather than ``perfect''.

scvalex 2009-08-10 07:08:31

Ok, you're right. So think of the title as "How to get as many Twitter links as possible?"

2009-08-10 11:19:42

Check out tweetSharp http://tweetsharp.com/. It's a .Net library that makes accessing the Twitter API much easier. I've used it in a little app to search Tweets for the name of the my company. There's also a dnrTV episode if you want to find out more.

I'm not quite sure what you mean by

"The normal API is limited too much"

You certainly don't have access to all tweets going back to the start, but as I understand it there's definitely enough in there to build a pretty decent Twitter client.

JustABitOfCode 2009-08-11 08:38:22

Thanks. But I'm not looking for a library. The problem is what API method to use and how to use it.

2009-08-11 18:22:59

Check out Twitterizer. It's way easier than any other Twitter ASP.NET library I've ever used.

Jason 2009-08-11 08:41:55

Thanks. But I'm not looking for a library. The problem is what API method to use and how to use it.

2009-08-11 18:23:38

so you downvoted me? uh, thanks.

Jason 2009-08-11 19:54:39

+3 A:

Twitter has a new Streaming API which it looks like anyone can access, it's still under alpha but this is what tweetmeme.com is probably using.

Tom 2009-08-11 12:31:21

Thank you. This is exactly what I was looking for. :)

2009-08-11 18:24:58

ansaurus

tags:

views:

answers:

How to get all Twitter links?

related questions