What's the best way to monitor a socket for new data and then process that data?

Question

Please pardon my C#.Net newbie status. If this is obvious and I missed it from the docs, a link to the relevant page or sample code would be appreciated.

I am working on an application that will accept a TCP socket connection from a Java application. (Yes, Java is required on that part. It's a Sun SPOT device and Java is the only option.) The Java app will be periodically writing new data to the socket, and my app's job is to take in the byte[], convert it to a string, process the data (update UI, etc), and possibly forward the data on to another computer running a similar C#.NET app.

Here's what I've done so far: Right now the app spins up a thread on launch that opens a socket. The Java app can successfully connect to the socket so that is working. I was looking at the NetworkStream's beginRead method and the dataAvailable, length, and CanRead properties, but I am not entirely sure how to ascertain when I've read one packet of data, usually about 512 bytes but could vary.

If the Java app writes data to the stream or there is a backlog of data (The Java app will be passing data rather quickly.) how can I make sure that I am reading only one packet of data at a time? If the Java app null terminates the data when it writes, will that help? Is it enough?

Lastly, the socket will only receive one connection, but I need to keep it open until there is an error or the connection is terminated. What is the most graceful way to handle this aspect? I don't think close and reopen with each data packet will work because of the rapid fire (nearly realtime) aspect of the Java app that is running on the Sun SPOT base station. Right now, when the base station terminates, my app dies a loud and painful death. :)

Thanks for reading and for any help you can offer.

Answer 1

When you are reading streaming data you need some sentinel/terminating character or a known size of the data to determine when to stop reading and process the information. Null characters or newlines are common. Another technique is using a fixed size header which specifies the length of the body.

The socket will remain open until you close it or the other side terminates, in which case you will get an error while reading from the socket, which you shuold handle. NetworkStream throws an IOException when the socket has been closed from the remote side during a read or write operation.

Answer 2

"If the Java app writes data to the stream or there is a backlog of data (The Java app will be passing data rather quickly.) how can I make sure that I am reading only one packet of data at a time?"

Be careful not to assume that you have any control over what data ends up in which packet. If you try to send the byte data { 'H', 'e', 'l', 'l', 'o' } there is no guarantee that all this data will be sent in a single packet. While it's extremely unlikely it is still possible that each packet could only contain a single byte so you'd receive all five bytes in 5 different events. The point being, do not rely on the packets in this manner. Instead, define your own End Of Message terminators and simply toss all incoming data into a byte buffer of some kind and have another function coming in detecting if there are any of these terminators present. If so read up to that terminator. So say for example you call the respective send method from your Java application twice containing the following data:

{ 'H', 'e', 'l', 'l', 'o', '\0' }
{ 'W', 'o', 'r', 'l', 'd', '\0' }

How your application should be prepared to receive the data should be something like this:

Server receives { 'H', 'e', 'l' }
Data stored in byte buffer { 'H', 'e', 'l' }
Check byte buffer for message terminator '\0'. None found. Buffer unchanged, no message processed.
Server receives { 'l', 'o', '\0', 'W' }
Data stored in byte buffer { 'H', 'e', 'l', 'l', 'o', '\0', 'W' }
Check byte buffer for message terminator '\0'. 1 found, extracted message { 'H', 'e', 'l', 'l', 'o' } and buffer updated { 'W' }

So while that wasn't exactly an answer to your original question I think it should give you a push in the right direction.

One thing you may run into is that there simply aren't any characters that couldn't be data instead of message terminators. For instance, many files contain the data \0 so these would wreck your message detection. How this is usually handled is by creating a header spec for your protocol and detecting whether you're expecting a header (In which case looking for \0 will denote the end of a message) or if you're waiting for a certain amount of data (Which could be specified by the last header received.) If this doesn't make sense and you think you might need to use this technique let me know and I'll add to this answer.

Answer 3

The way we've solved this is to have a combination of sentinel characters and fixed width fields. The first two bytes are a length header for the entire packet. Then, I read the packet into a byte[] buffer and then rely on our own packet structure to know that, for instance, the first two bytes are each to be interpreted as individual fields and then string fields are terminated with the \n character (0x0A if you're scoring at home). Then, long data fields are processed by reading in 8 consecutive bytes, etc. It seems to be working pretty well for us, but it's obviously a solution for a situation where one has control over both ends of the socket and not where one is only able to control the one end. Hope this helps someone else, too.