Check if Services/APIs have Expired Their Content - Best Practices for Optimizing/Caching Service Usage?

Question

If you want to build a website/mashup using 10+ services (twitter, facebook, linkedin, github, stackoverflow, etc.), and you also want to optimize your application...

Can you check if a URL's content is expired, without having to GET all of the pages content? If you could do that, then you could keep a local cache of their content in your system, and only update it every time their cache for that item changed. Would that work?

What is the recommended way for optimizing the use of services in this sense? I was just thinking about storing all my code snippets on Pastie and rendering them on my blog, but then I thought, well what about memcached and nginx? Everything would have to go through my Rails app if I did that! And that's a big performance hit. So can you optimize the use of services' content somehow?

Answer 1

I don't think there is a (reliable) way of checking to see if a URL is stale. You could try to replicate some of the methods here:

http://api.rubyonrails.org/classes/ActionController/AbstractRequest.html

Some that come to mind:

• if_modified_since
• fresh?

Hope this helps!