scrape a user's entire tweets

Question

I'd like to pull all of a user's tweets. I could do this the hard way (manually scraping twitter) or the easy way: using their api. The problem with the easy (api) way is that I seem to be limited to the 200 most recent tweets. What's a simple way to get all tweets? Thanks

Answer 1

Yes you can get up to 3,200 historical tweets by requesting as follows...

Make a request to:

http://api.twitter.com/1/statuses/user_timeline.format

And use the count parameter 200 and iterate through the page parameter from page 1 to 16 or until there are no more tweets.

Thats the only thing you can currently do because Twitter specifically say they prevent this in their API Doc...

https://apiwiki.twitter.com/Things-Every-Developer-Should-Know#6Therearepaginationlimits

I would add, please don't screen-scrape because it will cause undue load on Twitter and in bulk requests it would probably get your server blocked from accessing Twitter.

Answer 2

you can make sure you get all future tweets by subscribing to your Twitter RSS feed with Google Reader. Then you can use their infinite scrolling feature to look back to the first tweet tracked.