Sending email from a .net application via a mail server with a self-signed SSL certificate

Question

Hi,

I installed hmailserver 5.3.2 and configured it. It receives and sends emails normally, but I wanted to use it to send emails from a .net/C# application located in another server, and for that I wanted to use SSL communication. Before, the application was configured to send emails via gmail, on port 587 and it worked ok, but now we want to use our own mail server. We first configured the application to connect on smtp.domain.com on port 25 and that works, it sends the email.

Then we created a self signed certificate to test the if we could send the message through a secure channel.I created the certificate with openSSL setting common name as: mail.domain.com, smtp.domain.com, *.domain.com, domain.com. I opened port 587 on the firewall and configured hmailserver to use a certificate for inbound connections on that port. None of the certificates I created worked (I tried one and then created another one and so on), generating the following (generic) exception in the application:

System.Exception: _COMPlusExceptionCode = -532459699

Of course I also tried to connect via telnet: telnet smtp.domain.com 587, and I just got a blank screen. It is not a firewall issue since when I disable the ssl on port 587 I can connect normally. Looking at the log doesn't even show an attempt to connect when using 587 with SSL.

I already checked these questions: Getting SmtpClient to work with a self signed SSL certificate and Using a self-signed certificate with .NET’s HttpWebRequest/Response, but it didn't solve my problem. The approach with ServerCertificateValidationCallback didn't have any influence.

I tried with ports 25 (which is also proposed in one of the questions above), 465, 587, and with all 3 it happens the same: The initial handshake (SYN / SYN-ACK / ACK) and after about 80s the connection is closed (FIN), nothing in between.

Do I have to install that certificate somewhere so the .net application sees it as trusted? I mean, I already installed it as a Trusted Root Certification Authority and could check by running mmc, so I have no idea where to go now...

Thanks for the help!

PS: Not sure if this belongs to ServerFault since it concerns a C# application but also a mail server...

EDIT: Code sample:

ServicePointManager.ServerCertificateValidationCallback = 
(sender, certificate, chain, sslPolicyErrors) => true;

SmtpClient mailClient = new SmtpClient("smtp.domain.com");
mailClient.Credentials = new NetworkCredential("[email protected]", "pwd");
mailClient.Port = 587;
mailClient.EnableSsl = true;
MailMessage mailMessage = new MailMessage("mailAddressFrom", "mailAddressTo", "subject", "body");
mailMessage.IsBodyHtml = true;
mailClient.Send(mailMessage);

EDIT 2: Log (Based on Ramunas' suggestion):

"TCPIP" 3588    "2010-06-23 10:02:49.685"   "TCPConnection - Posting AcceptEx on 0.0.0.0:465"
"DEBUG" 3588    "2010-06-23 10:02:49.809"   "Creating session 24039"
"TCPIP" 772 "2010-06-23 10:04:29.639"   "TCPConnection - SSL handshake with client failed. Error code: 2, Message: End of file, Remote IP: X"
"DEBUG" 772 "2010-06-23 10:04:29.639"   "Ending session 24039"

Answer 1

This is a sophisticated mechanism but in simple words client (computer you're making connection from) should know about WHO is certificate issuer (in your case your server is certificate issuer). If it does not find it in it's Trusted Root Certificate Authorities list then it considers this connection to be unsafe. (I bet you've seen browser warning you about unsafe request to some https://.... site).

Open Certificates snap-in in your Microsoft management console on a client computer and try to add the same self signed certificate to a Trusted Root Certificate Authorities list.

Answer 2

I installed hMailServer, created self signed certificate, added it to hMailServer and was not able to send mail via it, too. Though I was successful while sending emails without certificate.

I enabled logging on hMailServer (for everything) and tried again with no luck. But I saw an error in a log file stating

"Severity: 2 (High), Code: HM5113, Source: TCPServer::Run(), Description: Failed to load certificate file. Path: <...>test.cer, Address: 0.0.0.0, Port: 25, Error: An invalid argument was supplied"

Maybe this is a case on your hMailServer also?