Just curious - what are some automatic or even semi-automatic techniques for reverse-engineering of communication protocols?
I am particularly interested in the case when one's sniffing traffic and trying to understand the protocol.
I could find a number of papers on scholar, but in my experience this is a completely manual process most of the times.
If anyone has experience in the field and feels like sharing it would be much appreciated.
Obtain some measure of control over a communication link and sniff the data. Then exercise the range of operations of the associated application to feel out how the protocol relates so you can gather general observations.
Google for the protocol. Maybe it is published. Maybe someone has already figured it out, or someone has carelessly leaked details about it.
Write a test program which replaces one end of the protocol: try eliciting responses from the other side by trial and error.
Often a protocol is a descendant of—or at least related in many ways to—another one. By seeing the specifics and having familiarity with many protocols, one can often make good educated guesses about its features and abilities.