Currently I use PHP mail(), but frequently this gives problems.
I'll bet you a lot of money it does NOT frequently give problems.
I have never come across problems in PHP's mail function. I have seen problems with bad php.ini settings for mail, and a host of problems with the programs which process mail after it has left PHP. Your use of the word "frequently" implies that it works some of the time, so, in the absence of frequent updates to php.ini, the problems are all on your mail handling infrastructure.
Indeed - I would recommend you go have a look at the PHP bug list - there will be lots of reports about problems getting mail from scripts to users inboxes - but none of them will be due to a failure of the PHP code.
So if the problem is elsewhere, using a replacement SMTP client will have no effect whatsoever (unless you configure it to bypass the bad MTA).
Understanding how email works, and why it fails, is far from trivial. When you add in to the mix, the lengths some people go to (usually undocumented) to prevent spam, it starts to become very complicated. Even if you had given precise details of your infrastructure and configuration, it would be difficult to even hazard a guess as to where why and how it is failing.
Certainly, you need to start by looking at your email logs and headers and checking your MTA and MUA configurations to start to resolve the problem.
C.