How do i fix my S3/memcache latency problem?

Question

We have SERIOUS latency issues SOMETIMES on our server.

We store 3 things of interest in S3 and stuff them into memcache as well.

• User avatars averaging ~25k
• text ~1.5k
• xml ~1.5k

we have dedicated 128meg of ram for memcached as of right now... as of right now it is riding 74 meg of it

doing some basic math we should easily be able to have around 30,000 text documents (with the xml representation of them) and 1,000 user avatars and still be under our 128meg dedicated to memcache

right now we have ~100 user avatars that could be pulled up at any given time we have in the hundred thousands of the text/xml documents but they don't get viewed like the avatars do...it's one here, on there type of thing

sometimes during the day the user avatars load super slowly (indicating they have to be loaded from s3) and other times (after they load of course) you can tell they are being served from memcached; same thing with the text documents

we are running merb under apache phusion with REE. we are using evan weaver's memcached gem built upon libmemcached-0.25.14 (which I fully understand is not the most up to date lib; this gem requires it)

From what I can see our latency issues are because of S3 which does have serious latency issues (sometimes 500ms for a single avatar). However, it seems that it should not be an issue considering it should be cached all the time. The default expiry on the cache is set to 1 week.

Relevant code is:

@cache = MMCACHE.clone
begin
  picture = @cache.get("/avatars/#{user.avatar}")
rescue
  picture = user.picture
  @cache.set("/avatars/#{user.avatar}", picture)
end
@cache.quit

the cloning/quitting is important since under apache/phusion will have problems sharing the connections when it forks it and if we didn't close out our connections they'd keep building up until we run out of file descriptors.

I'm starting to keep a much closer eye on memcache to see if I can track down my problems but any suggestions?? Should we just get rid of S3??

Answer 1

If I understand this correctly, you're storing image files in memcached backed by S3.

Why not just reference the images directly from S3, but set Expires HTTP headers on them so that they aren't pulled every time by the clients, this would have two advantages:

1. Pages would load more quickly as browsers would be pulling page components from multiple domains.

2. Simplification of your architecture.

Answer 2

Hi,

You can use Amazon CloudFront for static resources that will be fetched by client browser (like image, static html, css, javascript). CDN (Content Delivery Network) services eliminates the latency issues for this type of data; below is the description of the service:

Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments. Amazon CloudFront delivers your content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of your files. Like other Amazon Web Services, there are no contracts or monthly commitments for using Amazon CloudFront – you pay only for as much or as little content as you actually deliver through the service.

Regards, Sirmak