Good tools to understand / reverse engineer a top layer network protocol

Question

Hi Guys,

There is an interesting problem at hand. I have a role-playing MMOG running through a client application (not a browser) which sends the actions of my player to a server which keeps all the players in sync by sending packets back.

Now, the game uses a top layer protocol over TCP/IP to send the data. However, wireshark does not know what protocol is being used and shows everything beyond the TCP header as a dump.

Further, this dump does not have any plain text strings. Although the game has a chat feature, the chat string being sent is not seen in this dump as plain text anywhere.

My task is to reverse engineer the protocol a little to find some very basic stuff about the data contained in the packets.

Does anybody know why is the chat string not visible as plain text and whether it is likely that a standard top level protocol is being used?

Also, are there any tools which can help to get the data from the dump?

Answer 1

Best guess: encryption, or compression.

Even telnet supports compression over the wire, even though the whole protocol is entirely text based (well, very nearly).

You could try running the data stream through some common compression utilities, but I doubt that'd do much for you, since in all likelihood they don't transmit compression headers, there's simply some predefined values enforced.

If it's infact encryption, then you're pretty much screwed (without much, much more effort that I'm not even going to start to get into).

Answer 2

May be it uses encrypted data for security

Answer 3

It's most likely either compressed or encrypted.

If it's encrypted you won't have a chance.

If it's compressed you'll have to somehow figure out which parts of the data are compressed, where the compressed parts start and what the compression algorithm is. If your lucky there will be standard headers that you can identify, although they are probably stripped out to save space.

None of this is simple. Reverse engineering is hard. There aren't any standard tools to help you, you'll just have to investigate and try things until you figure it out. My advice would be to ask the developers for a protocol spec and see if they are willing to help support what you are trying to do.

Answer 4

If it's encrypted you do have a chance (in fact, you have a 100% chance if you handle it right): the key must reside somewhere on your computer. Just pop open your favorite debugger, watch for a bit (err, a hundred bytes or so I'd hope) of data to come in from a socket, set a watchpoint on that data, and look at the stack traces of things that access it. If you're really lucky, you might even see it get decrypted in place. If not, you'll probably pick up on the fact that they're using a standard encryption algorithm (they'd be fools not to from a theoretical security standpoint) either by looking at stack traces (if you're lucky) or by using one of the IV / S-box profilers out there (avoid the academic ones, most of them don't work without a lot of trouble). Many encryption algorithms use blocks of "standard data" that can be detected (these are the IVs / S-boxes), these are what you look for in the absence of other information. Whatever you find, google it, and try to override their encryption library to dump the data that's being encrypted/decrypted. From these dumps, it should be relatively easy to see what's going on.

REing an encrypted session can be a lot of fun, but it requires skill with your debugger and lots of reading. It can be frustrating but you won't be sorry if you spend the time to learn how to do it :)