About 5-6 years ago (the last time I looked into this sort of thing), I saw mailing list software in PHP using the mail()
function that was sending hundreds of messages everytime the "send to mailing list" feature was invoked. As the client added more and more names (into many thousands, last I checked) the system was getting rather slow. In the end, they bought some 3rd party software to handle large volume mailing and hosted on a server separate from their web server to avoid slowing down their web site.
As others have pointed out, you should clear this with your hosting provider before you start sending batches of more than a few dozen at a time - every hosting company will have their own policies, and if this violates the TOS, they can disconnect you/cut off your hosting. Ideally, large-volume mail transmission should be done from a server just for this purpose. That way, if it hangs or freezes, you won't have to worry about affecting other applications.
If you really are sending very very large amounts of mail, there are commercial packages out there that will also manage the mailing list, they will manage opt-outs and opt-ins, versions of emails, they will do text vs. HTML mail, etc... research some of them if you are serious.
I know this doesn't answer the main question of "alternatives to the mail()
function?" but it's the best I can do - I haven't seen any! The only thing I can think of is manually managing the SMTP connections in PHP (not sure how possible that is) or using some external library to do it.