A rails application in production should use smtp or sendmail to send mails?
Your question is incorrect - SMTP stands for Simple Mail Transfer Protocol whereas sendmail is the software piece to send the mail using this protocol.
Use sendmail.
SMTP is the protocol that is used by nearly all Internet hosts to send mail. This protocol is spoken by sendmail. Sendmail determines where to send your message and how.
Some mail programs (most, today) will connect directly to a mail server and speak SMTP to it. However, the "traditional" method - and arguable the better method - is to let sendmail do it.
There are two reasons for this: 1) nearly every program in UNIX that does what sendmail does is designed to be a drop-in replacement (this includes Postfix and Exim for instance); and 2) sendmail or its replacement was designed to handle mail and nothing else - by using sendmail you don't have to design a SMTP client.
The Mutt email client for UNIX is one email client that still refuses to talk SMTP directly to a mail server; a good (technical) description is on the wiki.
If you have a choice (on UNIX anyway) of talking SMTP directly or using sendmail, use sendmail - especially on servers.
They will both will work fine. Action Mailer supports both.
I have used SMTP on several projects successfully. My sense was that this was a little more "standard", but I may be wrong.
I haven't used sendmail. My concern would be that it may be harder to set up in a development environment if you aren't developing directly on Unix/Linux. Where you can talk directly to any SMTP server-- even a remote one,-- you would have to install sendmail on the rails machine to get it to work.
The main problem I run into with email is sending messages asynchronously. Without a local SMTP server, a local sendmail instance is going to be more performant.
Either way, it looks pretty easy to switch if you decide you picked wrong.
As NDP already mentioned, they both work fine - that is, if your volume of messages doesn't exceed a certain amount.
For example, if your application can talk SMTP to either the local SMTP server (on IP 127.0.0.1, Port 25) or a server in the same subnet (i.e., over a low-latency link), and that server does not use any content filters before it queues a message, you will usually be able to submit a lot more mails over SMTP in a shorter time.
A useful link for Postfix may be General Mail Delivery Performance Tips - note the quote saying
Submit mail via SMTP instead of /usr/sbin/sendmail.
However, on modern hardware, if you don't plan to submit more than about 10 messages per second, you shouldn't notice any real difference.