I made a server with SSL and blocking sockets. When I connect with telnet (so it does not do the handshake), the SSL_accept blocks indefinitely and blocks every new handshake/accept (and by definition new connections).
How can I solve this awful problem ?
Don't connect with telnet?
Normally, a service that can use either TLS or a plaintext connections works by establishing the connection in plaintext, then offering a command to request an "upgrade" of the connection to use TLS. Extended SMTP and the "STARTTLS" command is an example of this.
Services without such a command usually use distinct ports for TLS and non-TLS traffic. For example, HTTP on port 80 and HTTPS on port 443. A plaintext connection to port 443 isn't going to work.
What behavior would you like to see?
You can put the socket in non-blocking mode, then you'll get case SSL_ERROR_WANT_READ, or SSL_ERROR_WANT_WRITE from SSL_accept. You can then sleep a little, and try SSL_accept again. After some timeout value, you can quit and close the ssl and socket handles.
Note that this will affect all SSL operations, which means you'll need to do a similiar loop for all your read/write/shutdown calls. Basically, any call that can return WANT_READ or WANT_WRITE.
If you don't like the idea of polling, you can use select to figure out if you have data available on the socket...but that can get a bit complicated.
You can also try putting the socket back into blocking mode after the SSL_accept loop, and continue with your application.
Why not just set the socket stream to non-blocking mode before calling SSL_accept(), and then block on something like select() with a timeout if SSL_accept() returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE? Alternatively, you can block on select() before calling SSL_accept(). Either should work. That way you can at least bound the time the connection is blocked due to the DoS like behavior/attack.
Bear in mind that SSL/TLS is record-oriented, meaning you must loop until the full record is read. SSL_pending() can help in such cases.