Deciphering Encoding: Packet Analyzation Tools

Question

I am looking for better tools than wireshark for this. The problem with wireshark is that it does not format the data layer (which is the only part I am looking at) cleanly for me to compare the different packets and attempt to understand the third party encoding (which is closed source).

Specifically, what are some good tools for viewing data, and not tcp/udp header information? Particularly, a tool that formats the data for comparison.

To be very specific: I would like a program that compares multiple (not just 2) files in hex.

Answer 1

The problem is that data at the application layer is not standard (with very few exceptions, like HTTP, POP3 and so). Tools like Wireshark can't decode this info if they don't know the format.

I know some old versions of Ethereal (now Wireshark) had the option (you might have to enable it) to show the payload. Expect it to not make sense, most protocols are in binary!

Answer 2

Your best bet, frankly, is to roll your own.

Get a scripting language that you're comfortable with and start hacking away at the problem. First write up a simple multiway comparison, but as soon as you start finding patterns that you think are significant (or that you think might be) go back and add them to the code--eliminate them from the output, highlight them, translate them into another notation, replace them with their "meaning" or some high level description of their role--whatever seems appropriate. If you can't decide, make it an option.

You are asking for visualization software because you want something to help you form and internalize an understanding of their encoding. But the comparisons you are going to make are only part of the process you'll be using (which is essentially the scientific method)--you'll also be forming and revising guesses about what various parts of the packets mean, how they interact, etc.

No pre-built tool is going to help you there, but a good scripting language (say python, ruby, or even perl) will help a lot. As you form a theory, code it up and try it out. Mungle your code around, trying out different ideas as they occur to you, building up a bag of tricks customized to this problem as you go along.

-- MarkusQ

P.S. Don't fall into the trap of trying to use C or Java or something to do this. You'll be playing fast and loose, and should have a tool that doesn't require variable declarations, compilation, etc. There will be plenty of opportunity to tighten it up and rewrite it once you understand how it works.

Answer 3

For HTTP only, I used to used a great tool called Effetech.
(though looking at it now it seems they support more than just HTTP....)

Answer 4

If you have a capture you have done with ethereal you could read the capture, or you could do your captures with tcpdump. To capture use a tcpdump command like tcpdump -s 0 -qn -X , or tcpdump -X -r filename will read a capture.

22:08:33.513650 IP 192.168.32.10.40583 > 69.59.196.211.80: tcp 1261
0x0000:  4500 0521 18ec 4000 4006 322a c0a8 200a  E..!..@[email protected]*....
0x0010:  453b c4d3 9e87 0050 b0b6 4b4f 1598 0090  E;.....P..KO....
0x0020:  8018 1920 9b4f 0000 0101 080a 002e 701b  .....O........p.
0x0030:  093c bc38 4745 5420 2f75 7365 7273 2f32  .<.8GET./users/2
0x0040:  3032 3637 2f7a 6f72 6564 6163 6865 2048  0267/zoredache.H
0x0050:  5454 502f 312e 300d 0a48 6f73 743a 2073  TTP/1.0..Host:.s
0x0060:  7461 636b 6f76 6572 666c 6f77 2e63 6f6d  tackoverflow.com

Or there is another tcpick that may be what you want. You can capture the payload of tcp connections, and have it displayed as hex or saved.

Answer 5

Take a look at this blog post from BreakingPoint Labs. They are discussing manual protocol reverse engineering and a set of tools built on PacketFu to make this task easier.

Relevant to your question is the set of tools that makes it easy to identify and highlight changes to packets. Here's an example screenshot that shows the use of color to outline the packet differences:

Answer 6

Your problem is not to analyze network data, but compare binary files as you describe by your needs.

I would extract application data by means of any sniffer, i.e. by means of tcpdump used as described by Zoreadche, or Wireshark (i.e. by means of follow TCP session). Then save it to files and compare by any file comparation tool. You can try these (the most popular):

• Examdiff Pro. Really fast comparing directories.
• Winmerge. Though not as as fast as examdiff pro it is opensource and it is evolving rapidly. It is my first choice.
• Beyondcompare. This is the only file comparation tool I know that does exactly what you are looking for, that is comparing try files at the same time.