Best method of triggering a shell script from Java

Question

I have a shell script which I'd like to trigger from a J2EE web app.

The script does lots of things - processing, FTPing, etc - it's a legacy thing.

It takes a long time to run.

I'm wondering what is the best approach to this. I want a user to be able to click on a link, trigger the script, and display a message to the user saying that the script has started. I'd like the HTTP request/response cycle to be instantaneous, irrespective of the fact that my script takes a long time to run.

I can think of three options:

• Spawn a new thread during the processing of the user's click. However, I don't think this is compliant with the J2EE spec.
• Send some output down the HTTP response stream and commit it before triggering the script. This gives the illusion that the HTTP request/response cycle has finished, but actually the thread processing the request is still sat there waiting for the shell script to finish. So I've basically hijacked the containers HTTP processing thread for my own purpose.
• Create a wrapper script which starts my main script in the background. This would let the request/response cycle to finish normally in the container.

All the above would be using a servlet and Runtime.getRuntime().exec().

This is running on Solaris using Oracle's OC4J app server, on Java 1.4.2.

Please does anyone have any opinions on which is the least hacky solution and why?

Or does anyone have a better approach? We've got Quartz available, but we don't want to have to reimplement the shell script as a Java process.

Thanks.

Answer 1

For the second option, you can use a servlet, and after you've responded to the HTTP request, you can use java.lang.Runtime.exec() to execute your script. I'd also recommend that you look here : http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

... for some of the problems and pitfalls of using it.

Answer 2

My approach to this would probably be something like the following:

• Set up an ExecutorService within the servlet to perform the actual execution.
• Create an implementation of Callable with an appropriate return type, that wraps the actual script execution (using Runtime.exec()) to translate Java input variables to shell script arguments, and the script output to an appropriate Java object.
• When a request comes in, create an appropriate Callable object, submit it to the executor service and put the resulting Future somewhere persistent (e.g. user's session, or UID-keyed map returning the key to the user for later lookups, depending on requirements). Then immediately send an HTTP response to the user implying that the script was started OK (including the lookup key if required).
• Add some mechanism for the user to poll the progress of their task, returning either a "still running" response, a "failed" response or a "succeeded + result" response depending on the state of the Future that you just looked up.

It's a bit handwavy but depending on how your webapp is structured you can probably fit these general components in somewhere.

Answer 3

I'd go with option 3, especially if you don't actually need to know when the script finishes (or have some other way of finding out other than waiting for the process to end).

Option 1 wastes a thread that's just going to be sitting around waiting for the script to finish. Option 2 seems like a bad idea. I wouldn't hijack servlet container threads.

Answer 4

If your HTTP response / the user does not need to see the output of the script, or be aware of when the script completes, then your best option is to launch the thread in some sort of wrapper script as you mention so that it can run outside of the servlet container environment as a whole. This means you can absolve yourself from needing to manage threads within the container, or hijacking a thread as you mention, etc.

Only if the user needs to be informed of when the script completes and/or monitor the script's output would I consider options 1 or 2.

Answer 5

You mentioned Quartz so let's go for an option #4 (which is IMO the best of course):

• Use Quartz Scheduler and a org.quartz.jobs.NativeJob

PS: The biggest problem may be to find documentation and this is the best source I've been able to find: How to use NativeJob?

Answer 6

Is it necessary for your application to evaluate output from the script you are starting, or is this a simple fire-and-forget job? If it's not required, you can 'abuse' the fact that Runtime.getRuntime().exec() will return immediately with the process continuing to run in the background. If you actually wanted to wait for the script/process to finish, you would have to invoke waitFor() on the Process object returned by exec().

If the process you are starting writes anything to stdout or stderr, be sure to redirect these to either log files or /dev/null, otherwise the process will block after a while, since stdout and stderr are available as InputStreams with limited buffering capabilites through the Process object.

Answer 7

The most robust solution for asynchronous backend processes is using a message queue IMO. Recently I implemented this using a Spring-embedded ActiveMQ broker, and rigging up a producing and consuming bean. When a job needs to be started, my code calls the producer which puts a message on the queue. The consumer is subscribed to the queue and get kicked into action by the message in a separate thread. This approach neatly separates the UI from the queueing mechanism (via the producer), and from the asynchronous process (handled by the consumer).

Note this was a Java 5, Spring-configured environment running on a Tomcat server on developer machines, and deployed to Weblogic on the test/production machines.