How does Hudson work standalone on Linux?

Question

I need to know the mechanisms and flow for starting up Hudson, so I can figure out what's wrong. When you type in

nohup java -jar $HUDSON_WAR > $HUDSON_LOG 2>&1 &

(using real values for those variables, of course), what exactly happens? I've been a software engineer for 10+ years but mostly in embedded, and web services are out of my domain. I don't think it makes a difference, but I'm running Hudson on RHEL 5. It's not using tomcat or apache; it's running as a standalone service. What files are accessed and why?

Here are the specifics of the failure: I downloaded a plugin and after it had finished downloading I stopped and then restarted hudson, as I've done many times before. This time, however, when I try to access the Hudson main page, I get

Error 404
Exception:
Stacktrace:
 (none)

I suspected a permissions issue since I noticed some of the things in HUDSON_HOME were owned by root, so I recursively chown and chgrp'd that directory to make hudson own it all. That has not fixed the problem. I tried restoring the previous hudson.war but got the same behavior. One thing I notice is that right after I start it, if I go to the web page, I see Hudson's "starting up" page, and then it switches to the Error 404. I tried starting Hudson with a different port number and got the same behavior. I have no problem accessing a web interface to source control on the same server.

Update: I still suspect a permissions issue. It's possible the last person to restart Hudson did so as root.

Update[2]: Excerpt from hudson.log:
[Winstone 2010/10/21 16:05:17] - Beginning extraction from war file
hudson home directory: /home/hudson
[Winstone 2010/10/21 16:05:17] - HTTP Listener started: port=8080
[Winstone 2010/10/21 16:05:17] - AJP13 Listener started: port=8009
Using one-time self-signed certificate
[Winstone 2010/10/21 16:05:17] - Error starting listener instance
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:44)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:516)
    at winstone.Launcher.spawnListener(Launcher.java:232)
    at winstone.Launcher.<init>(Launcher.java:205)
    at winstone.Launcher.main(Launcher.java:391)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:600)
    at Main.main(Main.java:200)
Caused by: java.lang.NoClassDefFoundError: sun.security.x509.CertAndKeyGen
    at winstone.ssl.HttpsListener.<init>(HttpsListener.java:111)
    ... 12 more
Caused by: java.lang.ClassNotFoundException: sun.security.x509.CertAndKeyGen
    at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:77)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:385)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:653)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:619)
    ... 13 more

Answer 1

It seems that Hudson troubleshooting is normally done through the Manage Hudson interface using a Groovy script. Also, debug logging is most likely available. Quote from the "how to add loggers" page: "Tell us the symptom of your problem in the users list and we should be able to tell you where you need to look at. Also, this is really just a wrapper around the java.util.logging package, so if you program in Java, you might be able to guess what to look at." The user list that they're talking about is a mailing list.

How to add new/more loggers: http://wiki.hudson-ci.org/display/HUDSON/Logging
How to get onto the user list: http://wiki.hudson-ci.org/display/HUDSON/Mailing+List
Administration Details: http://wiki.hudson-ci.org/display/HUDSON/Administering+Hudson
Scripting: http://wiki.hudson-ci.org/display/HUDSON/Hudson+Script+Console
Some more scripting examples: http://community.jboss.org/wiki/HudsonHowToDebug

Answer 2

I'm just guessing here, but you may not be running the right Java binary.

Often on Linux, the java binary in /usr/bin is a gcc Java implementation or an OpenJDK package. I've had trouble in the past with these Java packages just not working well enough. If you have the SunOracle JDK installed, it's probably in /usr/java. I suggest setting JAVA_HOME to /usr/java/default and making sure the right java runtime is being run when you start Hudson.

export JAVA_HOME=/usr/java/default
nohup $JAVA_HOME/bin/java -jar $HUDSON_WAR > $HUDSON_LOG 2>&1 &

Personally, I think it's easier to use the init script provided with Hudson to start and stop Hudson. The script uses a central configuration file and makes sure Hudson is run as the right user.

sudo /etc/init.d/hudson start

Also, check HUDSON_LOG and/or /var/log/hudson/hudson.log for errors. If it's an issue with the plugin that you recently installed, you can disable the plugin by moving aside the [plugin] folder and [plugin].hpi file in HUDSON_HOME/plugins

Answer 3

Ultimately, this problem was caused by two issues: 1) Permissions were messed up. Since Hudson had previously been started as root, several of the files needed for operation became owned by root. I couldn't find a good synopsis of which hudson-related files ought to be owned by hudson and which as root and guessing wasn't working, so after making a backup, I re-installed Hudson.

2) Because I had been using the default hudson home of ".hudson" with a dot, but tried restarting with a specified HUDSON_HOME of /home/hudson (no dot), Hudson created a fresh home in /home/hudson and acted like it was a brand new installation. I had seen the HUDSON_HOME variable somewhere and thought it would be good to specify it in my startup script as /home/hudson, and thereby unwittingly changed the home used by Hudson. Once that was fixed, it's back to normal.