How can I reverse engineer scrambled packets in a windows app?

Question

I have a windows exe app that used to sends packets to a server in the clear. This app (lets call it the client app) is definitely close sourced, but some clever hacker hex-edited the binary, and made it send packets that are scrambled.

Now, obviously, those packets are scrambled in a way that is decipherable (otherwise the server would not be able to understand it), but what I wanted to do is to write an emulator that emulates this binary app, sending the same packets to the server, and being able to unscramble the response (if it is scrambled).

The hex-ed client required an extra dll in order to run, which the old client did not. I am assuming that somehow the hex-ed client managed to load that dll (lets call it client.dll) and the function of that dll is to implement the scrambling/unscrambling, by hooking into some windows api that rerouted all packets sent from the client.exe process.

If there are anyone who can direct me on how to even get started on working out how this all works, and how I can reverse engineer the scrambing, that would be really appreciated.

I have no idea what kind of information to provide, but if there is any lacking, just reply, and I will post with more details, and if anyone wants the binaries, I m happy to provide it.

---

binary download for any interested parties:

http://dl.getdropbox.com/u/46623/client.dll

http://dl.getdropbox.com/u/46623/newClient.exe

http://dl.getdropbox.com/u/46623/originalClient.exe

These wont run because the resource files are required - they are about 3 gigs, so too big to upload anywhere. Names have been changed to protect the guilty =) , but that probably doesnt protect the name of the dll...

Answer 1

You need to hook the functions exported by the additional DLL and look into the functions being called and the parameters being passed to them. This is not going to be easy since you do not have type information (e.g. the function signatures for the DLL exports.)

Look here for some information on API hooking. You will also need a good debugger try Windbg from microsoft.

As far as I can see the only option you have here is black box testing ie give known input to both systems and compare the responses against each other to find the differences and similarities.

               +--------------+
Input--------->| Original App |--------->Response1
               +--------------+

               +------------+
Input--------->| Modded App |--------->Response2
               +------------+

Now once you figure out how to use the functions from the additional dll you can use it yourself in the same way the original app does.

Answer 2

I'm assuming that the person which coded this hook which adds encryption to the packet I/O for the aforementioned program has either hooked the relevant Windows' socket APIs (WSASend, send, etc) or hooked the internal program functions used to send/receive data.

This being said, I'd suggest you use a hook detection program (e.g. RkUnhooker) to find out what is actually being hooked. Once you know what APIs are hooked you should also know where these hooks are going and from there on in you'll have to manually reverse engineer the hook functions.

As for the subject of learning how to do this, I couldn't direct you to just one tutorial to teach you everything but I highly suggest you look at the Tuts4You site, it has a plethora of tutorials which would meet all of your needs.

If possible, upload a copy of the edited client & the hook DLL, if I have the time I'll code you replica encryption & decryption functions.