Hi,
I've a setup with haproxy as frontend to two varnish servers (say srvA, srvB). srvA and srvB also have nginx running on 'em and varnish serves as frontend to these servers. Few days back we collected stats to find out how many requests being served by varnish cache (i.e. with two X-Varnish values) were getting timed out on haproxy. Interestingly we found that almost 30% of the response from varnish were getting timedout.
Following is a snippet of json log:
{"http":{"domain":"blah_blah_blah","request_method":"GET","request_protocol_version":"1.1","browser":"Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.3 (KHTML , like Gecko) Chrome/6.0.472.63 Safari/534.3","request_url":"/images/loading.gif","referrer":"http://blah_blah_blah","request_cookie":null, "return_code":200,"response_cookie":null,"byte_size":9754,"request_protocol":"HTTP"},"visitor":{"visit_unix":1286859600,"visit_unix_block_hour":1286859600,"geo":{"city":"blah" ,"region":"blah","organization":"blah","country_code":"blah","country_name":"blah","country_code3":"blah"},"visit_date":"2010-10-12 00:00:00","visitor_i p":"111.111.111.111","visit_date_block_hour":"2010-10-12 00:00:00","visitor_port":49800},"haproxy":{"listener_queue":0,"total_time_waiting_in_queue_ms":0,"feconn":1687,"haproxy_da emon_pid":5318,"total_time_to_establish_tcp_conn_ms":0,"beconn":1657,"listener_name":"slideshare","request_processing_time_ms":0,"srv_conn":32,"server_name":"blah_backend", "total_session_duration_time_ms":5013,"retries":0,"frontend":"srvA","termination_state":"cD--","haproxy_ip":"10.17.111.8","srv_queue":0,"total_time_to_receive_client_request_ms": 13,"actconn":1766,"haproxy_daemon":"haproxy"},"caching":{"x_varnish_cache_cur":2021094722,"x_varnish_cache_orig":2015028484,"x_varnish_cached":true},"app":{"x_bench_route_control ler":null,"x_bench_route_action":null,"x_bench_db_count":null,"x_bench_db_time":null,"x_bench_route":null}}
As the above snippet says, varnish sent back it's with two X-Varnish header values (2021094722 and 2015028484). However it resulted in connection in termination state on haproxy.
I am really confused about this. Please help me understand, why this can happen.
Thanks,
Mayank