The current application uses simple Java Mail to send couple emails a day but some of the emails never make it to the client.
Based on the application server logs there has been couple mail server time outs but that does not explain all the cases of the missing emails. Adding a retry feature would help with the time out problem but are there any other approaches to improve email reliability in general?
If you just want to send a couple of emails a day to a finite set of receivers and that's all, try to send them through a gmail account.
Set up a production quality mail server available to your application, and let it handle all the very dirty details of reliable mail sending. You are probably hitting some of the limitations like graylisting designed to keep spambots away.
A reasonably simple scenario would be Postfix on a Linux machine. Personally I like Ubuntu
It's the nature of SMTP that it does not implement transactional integrity.
About 6 years ago I did quite a detailled analysis of why mails from the company I was working at then were failing. I could only see as far as the receiving MTA, but this showed a very strong correlation between the type of MTA and the failure rate (at that time, Novell Groupwise and Sendmail at the remote end were the most reliable, MSExchange the least, with qmail and others in the middle). Note that this was highly empirical and may have reflected product choice vs available skills rather than intrinsic problems in specific MTAs - and its rather out of date now. Also, its not something you can effectively control.
Although, since you've got the opportunity to develop and implement your own logic on top of the MTA, there is no guarantee that:
1) if a message fails after leaving your MTA you will get any bounce notification back
2) if you send a message with a DSN request (see RFC 1891) that the remote system will actually send back the DSN
The most important things you can do to improve deliverability is to know a lot about SMTP, maintain your own MTA and configure it accordingly. One of the key problems these days is that everyone's trying to stop spam - and everyone's got their own method of doing that. And usually they won't tell you the recipe for their secret sauce. Indeed, with bayesian filtering, they may not even know!
I guess the next port of call (after you've checked your SPF is restrictive and published, and that you're not RBL'd) would be to look at how you establish if your mail is getting delivered - as I said, you can't rely on DSNs. Not can you rely on bugging your emails (e.g. by sending them out as HTML with, e.g. ) since most MUAs will not load remote content (again to prevent spam). Which just leaves the option of keeping the content serverside and sending out a clickable link to the original content. But this again assumes that your recipients always want to read your message.
C.
Thorbjørn and symcbean have both provided lots of useful information, but it may be overwhelming in its completeness. I'll try to make it more approachable:
About the worst thing you can do is build an SMTP client into your application and rely on that to send mail to somewhere in the world. A much better solution is to run a "standard" MTA and/or SMTP server on your own box locally or, at worst, inside your own network.
So your app only has to get mail as far as your own mail server, which will hopefully be on port 25 on the same machine. No SSL encoding, no spam filtering, not a bunch of things that can go wrong. Also, if your mail server is on the same machine as your app, they will (usually) either both be down or both up.
Once your app has pushed its mail into your local mail server (which is fast and nearly foolproof), it is that server's problem to get the mail sent to the final destination. On a Linux server you will have something like Sendmail, qmail, exim or postfix installed; on Windows, I don't know.
Any one of those "out of the box" mail servers is highly competent at getting mail out. An automatic repeat is already built in, with retries after (e.g.) 1 hour, 2 hours, 4, 12, 24 and 48 hours. Your mail server will try its darndest to deliver your mail, and will do so with no extra effort on your part. Failed attempts will show up in the mail server's log, and you can analyze that and draw your conclusions. If it fails after the last possible attempt, that is also noted in the log file, and you can conclude that something was wrong on the receiving side. All this power is already built in, and you shouldn't even think of trying to build it into a mail client of your own.
Final note: It's possible for the transfer to be successful physically, i.e. the message was delivered, but then it was treated as spam by the recipient's mail server or client; or the (human) recipient simply deleted it by accident. No software is going to solve that problem with certainty.