How do I go about reverse engineering a UDP-based custom game protocol with nothing other than Wireshark?

Question

How do I go about reverse engineering a UDP-based custom game protocol with nothing other than Wireshark? I can log a bunch of traffic, but then what? My goal is to write a dissector plugin for Wireshark that will eventually be able to decode the game commands. Does this seem feasible? What challenges might I face? Is it possible the commands are encrypted?

Answer 1

Commercial games expect that people will try to hack the protocol as a means to cheat, so will generally use encryption and probably tamper-detection as well.

Stopping this type of activity is of great concern to game makers because it ruins the experience for the majority of players when a few players have super-tools. For games like online poker the consequences are even more severe.

Answer 2

Yeah, it's feasible. But how practical it is will depend on the game in question. Compression will make your job harder, and encryption will make it impossible (at least through Wireshark - you can still get at the data in memory).

Probably the best way to go about this is to do it methodically - don't log 'a bunch of traffic' but instead perform a single action or command within the game and see what data is sent out to communicate that. Then you can look at the packet and try to spot anything of interest. Usually you won't learn much from that, so try another command and compare the new message with the first one. Which parts are in the same place? Which parts have moved? And which parts have changed entirely? Look especially for a value in a fixed position near the start of the packet that could be describing the message type. Generally speaking the start of the packet will be the generic stuff like the header and later parts of the packet will be the message-specifics. Consider that a UDP protocol often has its own hand-rolled ordering or reliability scheme and that you might find sequence numbers in there near the start.

Knowing your data types is handy. Integer values might be stored in big-endian or little-endian format, for example. And many games send data as floating point values, so be on the look-out for 2 or 3 floats in a row that might be describing a position or velocity.