SSL Handshake negotiation on Nginx terribly slow

Question

I am using Nginx as a proxy to 4 apache instances. My problem is that SSL negotiation takes a lot of time (600 ms). See this as an example: http://www.webpagetest.org/result/101020_8JXS/1/details/

Here is my Nginx Conf:

user www-data;
worker_processes  4;


events {
    worker_connections 2048;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log  /var/log/nginx/access.log;
    sendfile        on;
    keepalive_timeout  0;
    tcp_nodelay        on;
    gzip  on;
    gzip_proxied any;
    server_names_hash_bucket_size 128;


}

upstream abc {
     server 1.1.1.1 weight=1;
     server 1.1.1.2 weight=1;
     server 1.1.1.3 weight=1;


 }


server {
    listen   443;
    server_name  blah;

    keepalive_timeout 5;

    ssl  on;
    ssl_certificate  /blah.crt;
    ssl_certificate_key  /blah.key;
    ssl_session_cache  shared:SSL:10m;
    ssl_session_timeout  5m;
    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
    ssl_prefer_server_ciphers   on;

    location / { proxy_pass http://abc;

                 proxy_set_header X-Real-IP  $remote_addr;
                 proxy_set_header Host $host;
                 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    }

}

The machine is a VPS on Linode with 1 G of RAM. Can anyone please tell why SSL Hand shake is taking ages?