We're seeing ConnectException with message "Connection refused" sporadically appearing at times when connecting to our Tomcat server. Could these be caused by long GC cycles?
(We're activating up the relevant JVM flags to get more log messages on GC cycles, but I thought to ask this here anyways)
I highly doubt it. A run of the garbage collection would not cause your process to start refusing connections - if the runtime is busy in a GC phase then it means it's not actively running your code at the moment.
I think GC would never take so much time that network connections will time-out or get refused. I highly doubt that that is the problem.
We need more information on this.
Is this a Web Server that is refusing connections?
How much memory, which platform etc.
FWIW I have seen connections refused in a CORBA application during heavy GC. Note that this isn't the same as getting a Connection Refused message!
"connection refused" means that you reached to remote host but that it decided that you weren't allowed to connect. If GC was the problem, you'd get a "connection timed out" error.
It's possible but very, very unlikely. If you're thinking it's the GC, keep monitoring closely on various memory spaces from eden all the way to permgen and especially how they behave.
If I had to take an educated guess, I'd assume that your exhausting your connection pool because the actual operation(s) related to a single connection take a very long time to execute - and these would be the operations done before the initial connection is even answered, actions after that would cause a timeout.
The best way to find out is to have your app tell you how long GCs are taking. Startup with the option -verbose:gc and you'll get on STDOUT a record of the GC sweeps and how long they took. More intro GC stuff is at this summary
It's hard to tell without having any monitoring metrics from your VM, but it's not impossible that a gc run can cause connections to be refused, at least indirectly. What may happen, is that the gc run causes each request's process time to be increased or the requests to be temporarily interrupted, hence increasing the number of simultaneously required connections if the clients' request pattern stays the same.
For example, if you have 10 requests/second and each request takes 1 second to process, your server will at least on average cope with support for 10 concurrent connections. If the client(s) keep placing 10 requests/second to your server, but for some reason each request takes 2 seconds to process instead of 1, the server will have to support 20 concurrent connections or start refusing connection attempts if only a smaller number is allowed.
If I recall correctly the connection refused can happen if the operating system has too many connections which has been accepted but which needs to be processed. If this is indeed the case, Tomcat cannot service the sockets quickly enough.
My initial guess would be that you have something internally holding the threads up, so they cannot finish processing. Is your thread pool big enough? Do you have a globally synchronized object/method everything needs to passs through?
Please tell more about your situation.
Sporadic Connection Refused errors are often caused by sort of hiccup in the DNS or in the network hardware. To solve the first, try using IP address instead of hostname. To solve the second, ensure that the hardware is of high quality (i.e. no knacks in cables, no statical electricity in surrounding, etc).
If it was actually caused by the GC (which I don't believe, with such a thoroughly developed piece of software), then you would not be the only one who encountered this particular problem.
Oh, to be clear, a normal Connection Refused error actually means that the other side cannot be reached. This is often caused by wrong IP address, or a wrong port, or an unresolveable hostname or some software/hardware firewall in the network line which is blocking the connection.