Mysql/j Connector Unloads Mysteriously Under Tomcat 6/JDK 6

Hello - Thank you for taking time to look at my question.

I developing a Java WAR file intended for distribution to only one server at a time. So far, I have tested on 3 different servers with success, one Debian, one Ubuntu and the other, the Windows machine I'm developing on.

On the final deployment server, for some reason the app deploys fine, and upon each startup, runs well for about 3 minutes. Database connectivity works, everything runs fine. After 3 minutes however, I start seeing NoClassDefFoundError's consistently every time the app tries to access the DB.

I'm the following libraries with all of them packaged in the WAR file:

Mysql/j Connecter, latest version ActiveObjects, 0.8.2 Jasorb 1.3 Jython 2.5.1 (for some scripting) Tomcat 6.0.20 mod_python Apache

The error:

Exception in thread "Timer-4" java.lang.NoClassDefFoundError: com/mysql/jdbc/SQLError at com.mysql.jdbc.Util.handleNewInstance(Util.java:430) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:352) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:284) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:185) at net.java.ao.DatabaseProvider.getConnectionImpl(DatabaseProvider.java:771) at net.java.ao.DatabaseProvider.getConnection(DatabaseProvider.java:734) at net.java.ao.EntityManager.find(EntityManager.java:666) at net.java.ao.EntityManager.find(EntityManager.java:607) at net.java.ao.EntityManager.find(EntityManager.java:579) at com.sq.sa.SAPushAlertsTask.run(SAPushAlertsTask.java:32) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.SQLError at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:138 7) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:123 3) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) ... 13 more

Thank you for replying. I have tried removing the mysql/j connector from my WAR file and moving it to tomcat/libs. This produces a much larger stack trace that hopefully will be useful:http://pastie.org/653416Thank you so much!

samuraisam 2009-10-13 19:21:43

"this web application instance has been stopped already" message is highly suspicious - has you web app really been stopped (you can check that in tomcat's manager app)? If so, why? There are clearly some classloader issues there; the fact that moving your jdbc from war to tomcat/libs changed the behavior definitely points to that. What does your CLASSPATH contain (for user you're starting tomcat as)?

ChssPly76 2009-10-13 19:41:53

It get's more strange - The app isn't even *deployed*, and the MySQL/j connector isn't in the tomcat/lib directory however, the stack traces from the above post keep getting dumped to tomcat/logs/catalina.out. Is my tomcat installation borked? Thank you again for your help!

samuraisam 2009-10-13 19:56:24

You don't happen to have any _other_ apps deployed to that tomcat instance, do you? Are you hot-deploying your app? At least for the purposes of troubleshooting I'd suggest you install fresh tomcat instance, deploy your WAR and then start tomcat with **empty** CLASSPATH (make sure JAVA_HOME / path are appropriate as well). That would at least limit possible culprits to your webapp's lib. The other thing to check is jre/lib/ext - make sure there are no rogue jars there. Once you get that working you can see what's different for your current Tomcat install.

ChssPly76 2009-10-13 20:08:48

No, this is _supposed_ to be a fresh tomcat install (no other apps deployed either). I've tried all combinations of WAR/exploded war, hot loading/config loading... I'm guessing something was borked, so I'll try to re-install it myself and run with an empty CLASSPATH. Thank you so much for your help ChssPly76. I'll give your suggestions a try.

samuraisam 2009-10-13 20:24:11

You're welcome and good luck. Since your app works fine on other servers, this has got to be a classpath issue. Do post back here if your fresh tomcat reinstall with empty CLASSPATH still gives you trouble.

ChssPly76 2009-10-13 20:33:50

I still seem to be having the same issue. With a fresh tomcat install, I have been running by manually setting JRE_HOME and CLASSPATH like so: `JRE_HOME=/usr/local/java CLASSPATH='' /usr/local/tomcat/bin/startup.sh` It works for a few hours then starts throwing the same errors again - NoClassDefFoundError for SQLError. Here is the stack trace from a JSON client: http://pastie.org/655478

samuraisam 2009-10-15 01:36:53

Sorry to keep bugging you - but, any more ideas?

samuraisam 2009-10-15 01:43:13

Turned the logging *WAY* up on tomcat - and get the following output every second or so: http://pastie.org/655501

samuraisam 2009-10-15 02:00:34

I've looked at Connector/J source - there's really nothing strange about SQLError class - it's statically linked from Util (the next class in your stack trace). Couple things to try: write a two line servlet (or JSP) that throws an SQLError; deploy it to your Tomcat instance, see if it works. Try it in both your app context and another context. If it fails immediately you have a bad jar somewhere - if you're starting with empty classpath that leaves tomcat/libs and jre/lib/ext - check both. If it doesn't fail, it means your app is somehow getting undeployed - did you check it via manager?

ChssPly76 2009-10-15 02:32:45

I have checked it with manager - the path it's deployed under stays deployed. I just made the servlet you recommended, and down the rabbit hole we go... `SQLError e = new SQLError(); out.println(e.toString());` yields `com.mysql.jdbc.SQLError@1db8962` when I hit the URL in my app's context. If the app fails from here on out - I'll let you know. Perhaps there is a bug in ActiveObjects? Thank you again for your help!

samuraisam 2009-10-15 20:21:14

Really stretching here... perhaps I should build the WAR on a *nix machine instead of Windows?

samuraisam 2009-10-15 20:28:43

It definitely doesn't matter where the WAR gets built. The fact that you can get to SQLError initially means that at least at that point your classpath seems OK; I have the hardest time imagining how it could change after a while... It'd be interesting to run that test servlet once you get the error again and see whether it can or can't load SQLError at that point. I've never used ActiveObjects myself; are they loading jdbc driver via a _separate_ class loader perhaps? If so, that could explain some of the issues.

ChssPly76 2009-10-15 20:41:00

Another thing that throws me a little bit is `com.mysql.jdbc.NonRegisteringDriver` in your stack trace. Is there any reason why you're not using the regular `com.mysql.jdbc.Driver` driver?

ChssPly76 2009-10-15 20:43:25

It's possible that could be a side effect of using ActiveObjects. I haven't specified anything special in the configuration of ActiveObjects or JDBC besides supply a jdbc connection string to ActiveObjects. Is using tomcat's datasource functionality more portable/reliable?

samuraisam 2009-10-15 21:06:50

Tomcat datasource provides a level of indirection making it possible to change configuration post-deployment; it's not more reliable per se. However, cursory glance at ActiveObjects doc suggests that their default method to obtain connections is not using a connection pool which definitely is a problem in production. Plus the NonRegistering Driver seems rather iffy to me. Consider defining a pool yourself (using DBCP or C3P0 or any other) using regular com.mysql.jdbc.Driver and pointing your ActiveObjects configuration to that via appropriate PoolDriver.

ChssPly76 2009-10-15 21:23:50

ChssPly76 - thank you so much, again, for your help! I think not using a connection pool was my exact problem! It turns out, ActiveObjects will automatically try to use a connection pool if one is available. In my other deployments, hibernate and C3P0 were on the classpath, which ActiveObjects was automatically using as it's provider. On the production server, it created a new connection each request leading to a lot of weird errors - some of which I still can't explain.

samuraisam 2009-10-16 23:00:20

You're very welcome; glad I could help.

ChssPly76 2009-10-16 23:18:59

ansaurus

tags:

views:

answers:

Mysql/j Connector Unloads Mysteriously Under Tomcat 6/JDK 6

related questions