Problems with jetty crashing intermittently

Question

Hi, I'm having problems with jetty crashing intermittently, I'm using Jetty 6.1.24.

I'm running a neo4j Spring MVC webapp, Jetty will stay running for approx 1 hour and then I have to restart Jetty. It is running on small amazon ec2 instance, debian with 1.7gb of RAM.

I start Jetty using java -Xmx900m -server -jar start.jar

I am connecting to the server using putty, when Jetty crashes the putty session disconnects, I cannot see what error caused it to crash.

I would like to be able to see if it is an error generated by Spring, I'm not sure how to log the output from the spring app with Jetty. Or if it is Jetty or a memory issue, what would be the best way to monitor Jetty? I cannot recreate this on my local machine running windows. What do you think would be the best way to approach this? Thanks

Answer 1

This isn't really a programmer question; perhaps it'll be moved over to ServerFault.

You didn't specifically state which operating system you're using, but I'm hazarding a guess at some Linux distribution. You have two options of figuring out what's wrong:

1. Start your session in screen. Screen will live for as long as the actual machine is powered on, until you reboot the operating system (or you exit screen).

   you start screen like this

   screen
   

   and you get a new prompt where you can start your program (cd foo, jetty, etc). When you're happy and you just need to go somewhere, you can disconnect the screen by hitting CTRL+A and then CTRL+D. you'll drop back to the place you were before you invoked screen.

   To get back to seeing the screen you type screen -R which means to resume an existing screen. you should see jetty again.

   The nice thing is that if you lose connection (or you close putty by accident or whatever) then you can use screen -list to get a list of running screens, and then forcibly detach them -D and reattach them to the current putty -R, no harm done!

2. Use nohup. Nohup more or less detaches the process you're running from the console, so none of its output comes to the terminal. You start your program in the normal fashion, but you add the word nohup to your command.

   For example:

   nohup ls -l &
   

   After ls -l is complete, your output is stored in nohup.out.

Answer 2

When you say crash do you mean the JVM segfaults and disappears? If that's the case I'd check and make sure you aren't exhausting the machine's available memory. Java on linux will crash when the system memory gets so low the JVM cannot allocate up to its maximum memory. For example, you've set the max JVM memory to 500MB of which it's using 250MB at the moment. However, the Linux OS only has 128MB available. This produces unstable results and the JVM will segfault.

On windows the JVM is more well behaved in this scenario and throws OutOfMemoryError when the system is running low on memory.

1. Validate how much system memory is available around the time of your crashes.
2. Verify if other processes on your box are eating up a lot of memory. Turn off anything that could be competing with the JVM.
3. Run jconsole and connect it to your JVM. That will tell you how memory is being used in your JVM process and give you a history to look back through when it does crash.
4. Eliminate any native code you might be loading into the JVM when doing this type of testing.

I believe Jetty has some native code to do high volume request processing. Make sure that's not being used. You want to isolate the crashes to Java and NOT some strange native lib. If you take out the native stuff and find it works then you have your answer as to what's causing it. If it continues to crash then it very well could be what I'm describing.

You can force the JVM to allocate all the memory at startup with -Xms900m that can make sure the JVM doesn't fight with other processes for memory. Once it has the full Xmx amount allocated it won't crash. Not a solution, but you can easily test it this way.

Answer 3

When you start java, redirect both outputs (stdout and stderr) to a file:

Using Bash:

java -Xmx900m -server -jar start.jar > stdout.txt 2> stderr.txt

After the crash, inspect those files.

If the crash is due to a signal (like SEGV=segmentation fault), there should be a file dump by the JVM at the location you've started java. For Sun VM (hotspot), it's something like hs_err_pid12121.log (here 12121 is the process ID).

Answer 4

Putty disconnecting STRONGLY hints that the server is running out of memory and starts shutting down processes left and right. It is probably your jetty instance growing too big.

The easiest thing to do now, is adding 1-2 Gb more swap space and do it again. Also note that you can use the jvisualvm to attach to the jetty instance to get runtime information directly.