First, let me recommend that you do not rely on the DLQ for this. The DLQ is a QMgr-level resource that is used to store messages that the channel cannot resolve. Because it is a potential attack vector, most security-conscious shops will not grant access to applications, or if they do it is put-only access.
The best way to do this is to create an app-specific exception queue. If the app has several input queues, they can all use the same exception queue and then the application support team can manage any messages that land there without the security issues.
Whichever way you go, getting the app to honor this is extremely easy. For example, suppose your app is reading from JAX.SVC.REQUEST. You define the exception queue and point the BOQ* fields at it:
DEF QL(JAX.SVC.EXCEPTION)
ALTER QL(JAX.SVC.REQUEST) BOQNAME(JAX.SVC.EXCEPTION) BOQTHRESH(5)
The JMS classes will inquire on the BOQ* attributes of the queue when it is opened and check the backout count of each message read. In your program use a transacted session and if you can't process the message, call the session.backout() method. The message will be read and backed out no more than BOTHRESH times, then requeued to the queue named in BOQNAME. If that queue is full or unavailable the DLQ will be tried. If that fails, the classes will throw an exception.
We pick a BOTHRESH > 1 to allow for the possibility that a processable message will be backed out if, for example, the QMgr is shut down.
I usually trigger the exception queue so that it can raise an alarm or send an email if something lands on it. If you have a monitoring tool you can check for depth > 0 instead.
If for some reason yo don't want to use the JMS functionality to automatically requeue the message, the application would need logic to requeue it. If you were to place the message on the DLQ you will want to prepend a DLQ header. Failure to do so will probably break the DLQ handler or any other instrumentation watching that queue.