I've written a mail app or three in my days. Can't really comment on swiftmailer at all, but some things to keep in mind:
1) Lists of this scale are long-running operations. So long-running that they really should not be handled by web processes at all but rather as some sort of batch job.
2) Something sometime is going to screw up and cut off the stream of stuff to send. Which means your mailer app should be able to a) know what it is doing and b) be able to restart itself at any point.
3) Related to #2--have lots of instrumentation and logging on this puppy. You will need to be able to figure out when this long-running batch process screwed up and why at some point.
So, basically, the key thing to focus on is making the batch job bits work. Exactly what email sending engine is almost a secondary concern.