JSPs should be used in the presentation layer, servlets for business logic and back-end (usually database layer) code.
I don't know any reason why you can't use a JSP as you describe (it gets compiled to a servlet by the containter anyway), but you're right, the preferred method is to make it a servlet in the first place.
Yeah, this should be a servlet. A JSP may be easier to develop, but a servlet will be easier to maintain. Just imagine having to fix some random bug in 6 months and trying to remember how it worked.
JSP's are essentially markup that automatically gets compiled to a servlet by the servlet container, so the compile step will happen in both instances. This is why a servlet container that supports JSP must have the full JDK available as opposed to only needing the JRE.
So the primary reason for JSP is to reduce the amount of code required to render a page. If you don't have to render a page, a servlet is better.
Most java applications nowadays are build on the MVC pattern... In the controller side (servlet) you implement business logic. The servlet controller usually forward the request to a jsp that will generate the actual html response (the View in MVC). The goal is to separate concerns... Thousands of books have been written on that subject.
JSPs: To present data to the user. No business logic should be here, and certainly no database access.
Servlets: To handle input from a form or specific URL. Usually people will use a library like Struts/Spring on top of Servlets to clear up the programming. Regardless the servlet should just validate the data that has come in, and then pass it onto a backend business layer implementation (which you can code test cases against). It should then put the resulting values on the request or session, and call a JSP to display them.
Model: A data model that holds your structured data that the website handles. The servlet may take the arguments, put them into the model and then call the business layer. The model can then interface with back-end DAOs (or Hibernate) to access the database.
Any non-trivial project should implement a MVC structure. It is, of course, overkill for trivial functionality. In your case I would implement a servlet that called a DAO to update the status, etc, or whatever is required.
JSPs are a shortcut to write a servlet. In fact they are translated to servlet java code before compilation. (You can check it under some tomcat subdir wich I don't remember the name).
To choose between servlet an JSP I use a simple rule: if the page contains more html code than java code, go for JSP, otherwise just write a servlet. In general that translates roughly to: use JSPs for content presentation and servlets for control, validation, etc.
Also, Its easier to organize and structure your code inside a servlet, since it uses the plain java class syntax. JSPs tend to be more monolithic, although its possible to create methods inside then.
A JSP is compiled to a servlet the first time it is run. That means that there's no real runtime difference between them.
However, most have a tradition to use servlets for controllers and JSPs for views. Since controllers are just java classes you can get full tool support (code completion etc.) from all IDEs. That gives better quality and faster development times compared to JSPs. Some more advanced IDE's (IntelliJ IDEA springs to mind) have great JSP support, rendering that argument obsolete.
If you're making your own framework or just making it with simple JSPs, then you should feel free to continue to use JSPs. There's no performance difference and if you feel JSPs are easier to write, then by all means continue.
In an MVC architecture, servlets are used as controller and JSPs as view. But both are technically the same. JSP will be translated into servlet, either in compile time (like in JDeveloper) or when accessed for the first time (like in Tomcat). So the real difference is in the ease of use. I'm pretty sure that you'll have a hard time rendering HTML page using servlet; but opposite to common sense, you'll actually find it pretty easy to code even a fairly complex logic all inside JSP (with the help of some prepared helper class maybe). PHP guys do this all the time. And so they fall into the pitfall of creating spaghetti codes. So my solution to your problem: if you found it easier to code in JSP and it wouldn't involve too many code, feel free to code in JSP. Otherwise, use servlet.
Agreed with all the points above about the differences between JSPs and Servlets, but here are a couple additional considerations. You write:
I have an application that sends the customer to another site to handle the payments. The other site, outside of the customer, calls a page on our server to let us know what the status is of the payment. The called page checks the parameters that are given by the payment application and checks to see whether the transaction is known to us. It then updates the database to reflect the status. This is all done without any interaction with the customer.
Your application is consuming the payment service of another application. Your solution is fragile because if the payment service in the other application changes, that breaks your JSP page. Or if you want to change your application's payment policies, then your page will have to change. The short answer is that your application should be consuming the application's payment service via a web service. Neither a servlet nor a JSP page is appropriate place to put your consumption logic.
Second, along those lines, most usages of servlets/JSP pages in the last few years have been put inside the context of a framework like Spring or Struts. I would recommend Spring, as it offers you the full stack of what you need from the server pages to web service gateway logic to DAOs. If you want to understand the nuts and bolts of Spring, I would recommend Spring in Action. If you need to understand better how to tier an enterprise architecture written in a language like Java (or C#), I would recommend Fowler's Patterns of Enterprise Application Architecture.
I know this isn't the popular answer today, but: When I'm designing an app from scratch, I always use JSPs. When the logic is non-trivial, I create ordinary Java classes to do the grunt work that I call from the JSP. I've never understood the argument that you should use servlets because, as pure Java classes, they are more maintainable. A JSP can easily call a pure Java class, and of course an ordinary Java class is just as maintainable as any servlet. It's easier to format a page in a JSP because you can put all the markup in-line instead of having to write a bunch of println's. But the biggest advantage of JSPs is that you can just drop them in a directory and they are directly accessible: you don't need to mess with setting up relationships between the URL and the class file. Security is easily handled by having every JSP begin with a security check, which can be a single call statement, so there's no need to put security into a dispatch layer.
The only reason I can see to use a servlet is if you need a complex mapping between URLs and the resulting execution class. Like, if you want to examine the URL and then call one of many classes depending on session state or some such. Personally I've never wanted to do this, and apps I've seen that do do it tend to be hard to maintain because before you can even begin to make a change you have to figure out what code is really being executed.
There are 2 pretty simple rules: