What framework + application server + development environment to use for Java services?

Question

I'm trying to write a Java service which runs 24/7, scrapes content from the web, and stores it into a database. What is the best framework to use for this given that I'd like to...

1.) Have an application server that I can deploy my code to (and have it run automatically). This application server should sit on a separate box from the machine that my development environment will run on.

2.) Have a development platform (I would prefer something based on eclipse) which allows me to deploy my code directly to the application server (so I don't have to ftp everything over to test).

3.) Utilize a framework like Spring.

In effect, I'd like to know what to choose for my...

1.) application server

2.) development environment (ide) -- if eclipse, what server adapter to use

3.) framework

So far, I've tried using Virgo with SpringSource STS, but was unable to configure the web server adapter for a non-localhost-residing server. I don't want to have to install Virgo on my development box, and I don't want to have to ftp hop my code over to my production server in order to deploy.

Answer 1

Why don't you just use Tomcat or some other web container, but it may be better to split this into two applications.

Have one that goes out and does the scraping, as a standalone application, for this you can pick anything, I would go with Groovy (http://groovy.codehaus.org/), as ease of development and maintenance is important here, and you can use the Groovy plugin for Eclipse.

The other would be the web service and for this I would think Scala (http://www.scala-lang.org/) would be nice, if you have time to learn it, but Grails (a groovy framework) would be beneficial, so you can write a REST or SOAP web service.

By separating them then you can pick the best solution for that particular aspect, since the web server shouldn't be involved in scraping, but the web server will want to read from the database.

These two languages run on the JVM and can use regular java classes/libraries, but there are improvements over plain Java in them.

Answer 2

Turns out there are some maven plugins that will remotely deploy my app for me. The most notable is Cargo. This way, I can keep all of my initial tools/services the same (Virgo, STS, Maven).

Answer 3

I've actually build something similar quite recently. My application could run without a servlet container or an application server. The reason I choose to run my app in a Tomcat servlet engine is so that I can add a REST API to it to easily retrieve server status information, but I digress.

The plain vanilla Eclipse J2EE installation has decent Tomcat support so without knowing more about your tastes and specifics I'd go with that.

To make your application self starting you need to implement the ServletContextListener interface:

public class ServerClass extends HttpServlet implements ServletContextListener {

  public void contextInitialized( ServletContextEvent event ) {

    // create and start a thread here.

  }

  public void contextDestroyed( ServletContextEvent event ) {

  }

}    

Add the following to your web.xml:

<listener>
    <listener-class>com.my.ServerClass</listener-class>
</listener>

Which framework you want to use, only you can decide. Your question is to generic to give a decent answer on that. Read up on a few and pick one. Plain old Java will also do just fine and otherwise Scala might be a good substitute choice.

So, to answer your questions:

1. Tomcat servlet engine
2. Vanilla J2EE Eclipse version
3. Plain old Java