How do I get around the Twitter API caching problem?

Question

I'm building a Twitter app that requires to check user data somewhat frequently, but I'm facing trouble with a cache that's oddly on Twitter's side, not mine.

Try the following user:

users/show in XML: http://twitter.com/users/show.xml?screen_name=technolocus

users/show in JSON: http://twitter.com/users/show.json?screen_name=technolocus

normal page: http://twitter.com/technolocus

All these methods of accessing data should return the same values, right? Check the statuses_count for each of them.

XML: 12548

JSON: 12513

normal: 12498

The normal method (i.e. just visiting the profile non-programatically) serves up the most correct value of 12498. If I post or delete tweets to this account, it gets updated on the profile page instantly, but the XML and JSON methods still return cached data.

At this point, the values of the XML and JSON methods are 12 to 18 hours old respectively.

I first tried to access these methods from my website (hosted on Dreamhost). I thought it was Dreamhost caching the responses. Then I tried to access the API directly from my browser. I did a cURL from the command line from my machine after that. It wasn't dreamhost. I thought it was probably my ISP (I think they use NetApp or something like that). Then I asked a friend in another corner of India to try it. He's getting the exact same cached responses as I am.

So it isn't Dreamhost's cache; it isn't my ISP or my country's cache. There's only one conclusion - Twitter is caching responses.

How in the heavens do I get around this?!?

Forgot to mention this: The script on the server is in PHP and is using cURL to retrieve the XML and JSON data from Twitter, while the local tests have been just using the browser. Both have the exact same result!

Answer 1

First, I think you should report this a a bug to Twitter. I see the same discrepancy as you, and no matter what that seems like a bug. Even if they're caching, I'd expect that a cache on their side would store an abstract form that would then be rendered into HTML, JSON, and XML. I wonder if what's actually going on is that these requests are performing similar but different queries.

Are you sure that the values are "old"? For example, did you actually delete about 50 updates recently (since you say the HTML one is newest but shows a lower count than the other two)? If you create another update do you see the HTML number increment while the other numbers stay the same, or do they all increment simultaneously?

Answer 2

If what you are saying is accurate, and it probably is, generally, you can't get around it. Twitter would want to be caching its responses since they are costly to reproduce every single time.

When you use Twitter's APIs, you end up being bound by its conventions, even if that includes caching.

Your best bet is to tweet to @twitterapi and get them to give you a response as to why the two representations are divergent.