How to build message from tcp segments.

Question

Hi,

working in C#, I use SharpPCap to get segments from a winpcap trace.

I need to rebuild all the messages sent and received in that trace.

In my situation, the client's and server's IP will never be the same. Client's port does not necessarily change.

The protocol used by the message could be HTTP or something custom that I don't know.

That's how I currently do it :

            if (ipPacket.Protocol == IPProtocolType.TCP)
            {
                TcpPacket tcpPacket = (TcpPacket)ipPacket.PayloadPacket;

                Packet dataPacket = tcpPacket;
                while (dataPacket.PayloadPacket != null)
                    dataPacket = dataPacket.PayloadPacket;

                if (dataPacket.PayloadData.Length > 0)
                {
                    if (m_MessageContainer.IsEmpty()
                        || ((m_MessageContainer.Last().SourceIp.ToString() != ipPacket.SourceAddress.ToString())
                             && tcpPacket.Psh))
                    {
                        m_MessageContainer.Add(BuildMessage(ipPacket, tcpPacket));                      
                    }
                    m_MessageContainer.Last().AddData(dataPacket.PayloadData);
                }
            }

The problem with my solution is when the client send two request in a row. I just merge the two messages in one. If I change

if (m_MessageContainer.IsEmpty()
  || ((m_MessageContainer.Last().SourceIp.ToString() != ipPacket.SourceAddress.ToString())
       && tcpPacket.Psh))
 {
     m_MessageContainer.Add(BuildMessage(ipPacket, tcpPacket));                      
 }

by

if (m_MessageContainer.IsEmpty()
  || tcpPacket.Psh)
 {
     m_MessageContainer.Add(BuildMessage(ipPacket, tcpPacket));                      
 }

then a problem occurs when a message is split between more than one tcp segments and the flag psh is set on at least two of those tcp segment.

I need a way to correctly merge segments to rebuild original messages. I can't rely on the protocol used over TCP.

Thank you!

Edit : In wireshark, when you do follow tcp stream, it doesn't necessarily know the protocol over tcp but it is able to show each request and response in different colors. How does it is able to do that? I am seeking the same functionality because in my situation, there will never be a second request before a response is received in a stream. Thanks

Answer 1

There's no way to know just from observing the stream on the wire just how the original application code wrote to the socket. You are talking about using hints that might show up under certain circumstances but you can't rely on them (as you have discovered).

write(sock, "GET / HTTP/1.0\r\n\r\n", len);

write(sock, "GET / HTTP/1.0\r\n", len);
write(sock, "\r\n", 2);

write(sock, "GET / HTTP/1.0\r\n", len);
sleep(1);
write(sock, "\r\n", 2);

All three of those examples are legal ways to write an HTTP query. All three could look the same on the wire. Certainly the last one has a strong chance of being different even though it means the same thing (imagine the sleep isn't explicit, but perhaps caused by reading cookies off of disk). Even so the last one could be folded into a single transmission if the particular TCP socket is being held off by retries at the moment the two writes occur. The second example might show up as one packet or two just based on the other load on the interface (or socket options like CORK).