Realtime Twitter Replies?

Question

I have created Twitter bots for many geographic locations. I want to allow users to @-reply to the Twitter bot with commands and then have the bot respond with the results. I would like to have the bot reply to the user as quickly as possible (realtime).

Apparently, Twitter used to have an XMPP/Jabber interface that would provide this type of realtime feed of replies but it was shut down.

As I see it my options are to use one of the following:

REST API

This would involve polling every X minutes for each bot. The problem with this is that it is not realtime and each Twitter account would have to be polled.

Search API

The search API does allow specifying a "-to" parameter in the search and replies to all bots could be aggregated in a search such as "-to bot1 OR -to bot2...". Though if you have hundreds of bots then the search string would get very long and probably exceed the maximum length of a GET request.

Streaming API

The streaming API looks very promising as it provides realtime results. The API allows you to specify a follow and track parameters. follow is not useful as the bot does not know who will be sending it commands. track allows you to specify keywords to track. This could possibly work by creating a daemon process that connects to the Streaming API and tracks all references to the bot's names. Once again since there are lots of bots to track the length and complexity of the query may be an issue. Another idea would be to track a special hashtag such as #botcommand and then a user could send a command using this syntax @bot1 weather #botcommand. Then by using the Streaming API to track all references to #botcommand would give you a realtime stream of all the commands. Further parsing could then be done to determine which bot to send the command to. This blog post has more details on the Streaming API

Third-party service

Are there any third-party companies that have access to the Twitter firehouse and offer realtime data?

I haven't investigated these, but here are a few that I have found:

• Gnip
• Tweet.IM
• excla.im
• TwitterSpy - seems to use polling, not realtime
• tweethook

I'm leaning towards using the Streaming API. Is there a better way to get near realtime @-replies for many (hundreds) of Twitter accounts?

UPDATE: Twitter just announced that in the future they will have User Streams which expands upon the Streaming API. User Streams Preview

Answer 1

There's the twitter firehose but you're probably best off using the Streaming API. The firehose is open to Google (try googling your twitter name) and as the link says they're opening it up to all soon enough.

You'll want to get your IP whitelist too.

If your not already, you want to check out the GoogleGroup for twitter devs.

Answer 2

The track predicate for the streaming api would actually be useful because if you follow your bot's user IDs, you'll get all the messages made by your bots and all the other messages that mention your bots @usernames (including @replies). It really does track everything public on twitter relating to the user IDs you follow with it, give it a shot.

Answer 3

Either track or follow will work for the cases you describe. See http://apiwiki.twitter.com/Streaming-API-Documentation#track for details on what track actually does. The doc on follow is on the same page.

There are rate limits of sorts on the streaming API, but they have to do with how big a slice of the total tweet stream you're consuming. For writing a bot like this you won't hit these limits without a pretty big user base. And when you get that user base you can apply for elevated access levels that increase the rate limets.

Answer 4

REST API:

The most comprehensive results with the least amount of false positives. Will include protected statuses if the bot is following the protected account. If you poll every thirty seconds it is pretty close to realtime and you will be well under your rate limit (350/hour) if you are using api.twitter.com/1 with OAuth.

Streaming API:

You will want to avoid the Search API. It is trending more and more towards popular results and not complete results.

Streaming API

The fastest but also likely to miss some statuses as well as include false positives. Protected statuses for example are not included. Track for a screen_name will return statuses with that screen_name in it but will also include tweets that just have the screen_name as a string without the @ so be sure to filter on your side.