How to detect an incoming SSL (https) handshake (SSL wire format)?

Question

I'm writing a server which is accepting incoming TCP connections. Let's suppose the server has accepted a TCP connection, and has already received 16 (or so) bytes from the client. Knowing those 16 bytes how can the server detect whether the client wants to initiate an SSL handshake?

I've made an experiment, which showed that on my Linux system connecting to localhost (either 127.0.0.1 or AF_UNIX) via SSL makes the client send the following handshake (hexdump), followed by 16 seemingly random bytes:

8064010301004b0000001000003900003800003500001600001300000a07
00c000003300003200002f03008000000500000401008000001500001200
0009060040000014000011000008000006040080000003020080

How should the server probe these first few bytes, just to be able to determine whether the client is sending an SSL handshake? The probe must return true for all valid SSL handshakes, and it must return false with high probability for a message sent by the client which is not an SSL handshake. It is not allowed to use any libraries (like OpenSSL) for the probe. The probe must be a simple code (like a few dozen lines in C or Python).

Answer 1

I could figure this out based on the implementation of the ClientHello.parse method in http://tlslite.cvs.sourceforge.net/viewvc/tlslite/tlslite/tlslite/messages.py?view=markup

I am giving two solutions here in Python. IsSSlClientHandshakeSimple is a simple regexp, which can yield some false positives quite easily; IsSslClientHandshake is more complicated: it checks the consistency of lengths, and the range of some other numbers.

import re

def IsSslClientHandshakeSimple(buf):
  return bool(re.match(r'(?s)\A(?:\x80[\x0f-\xff]\x01[\x00-\x09][\x00-\x1f]'
                       r'[\x00-\x05].\x00.\x00.|'
                       r'\x16[\x2c-\xff]\x01\x00[\x00-\x05].'
                       r'[\x00-\x09][\x00-\x1f])', buf))

def IsSslClientHandshake(buf):
  if len(buf) < 2:  # Missing record header.
    return False
  if len(buf) < 2 + ord(buf[1]):  # Incomplete record body.
    return False
  # TODO(pts): Support two-byte lengths in buf[1].
  if ord(buf[0]) == 0x80:  # SSL v2.
    if ord(buf[1]) < 9:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    if ord(buf[3]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[4]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    cipher_specs_size = ord(buf[5]) << 8 | ord(buf[6])
    session_id_size = ord(buf[7]) << 8 | ord(buf[8])
    random_size = ord(buf[9]) << 8 | ord(buf[10])
    if ord(buf[1]) < 9 + cipher_specs_size + session_id_size + random_size:
      return False
    if cipher_specs_size % 3 != 0:  # Cipher specs not a multiple of 3 bytes.
      return False
  elif ord(buf[0]) == 0x16:  # SSL v1.
    # TODO(pts): Test this.
    if ord(buf[1]) < 39:  # Message body too short.
      return False
    if ord(buf[2]) != 0x01:  # Not client_hello.
      return False
    head_size = ord(buf[3]) << 16 | ord(buf[4]) << 8 | ord(buf[5])
    if ord(buf[1]) < head_size + 4:  # Head doesn't fit in message body.
      return False
    if ord(buf[6]) > 9:  # Client major version too large. (Good: 0x03)
      return False
    if ord(buf[7]) > 31:  # Client minor version too large. (Good: 0x01)
      return False
    # random is at buf[8 : 40]
    session_id_size = ord(buf[40])
    i = 41 + session_id_size
    if ord(buf[1]) < i + 2:  # session_id + cipher_suites_size doesn't fit.
      return False
    cipher_specs_size = ord(buf[i]) << 8 | ord(buf[i + 1])
    if cipher_specs_size % 2 != 0:
      return False
    i += 2 + cipher_specs_size
    if ord(buf[1]) < i + 1: # cipher_specs + c..._methods_size doesn't fit.
      return False
    if ord(buf[1]) < i + 1 + ord(buf[i]): # compression_methods doesn't fit.
      return False
  else:  # Not SSL v1 or SSL v2.
    return False
return True