Designing a network protocol for realtime data / mobile devices

Question

I'm faced with a following dilemma:

Design a new network protocol which would be used between a server (Java software) and desktop and mobile clients. The mobile clients include J2ME, Android and maybe in the future even iPhone.

The data stream is a realtime, constant stream with also more infrequent parts. The clients show waveforms of this data and also data which doesn't need to be updated instantly. The clients should also be authenticated.

I'd like to avoid creating a totally custom TCP protocol implementation from scratch if possible.

These days people usually recommed doing everything in REST style which I also really like. In this case I'm a bit hesitant though: how would you implement a constant stream of data on top of REST? A chunked HTTP response?

I'm also considering non-plaintext protocols (the current ones which I am replacing are binary protocols). Those current protocols have their rather serious issues so they really should be replaced.

Google protocol buffers looks like a pretty strong candidate for handling the low-level details, but I'm not sure if it can be used from Android. And I'm pretty sure that the iPhone implementation would have issues with it as well.

There's also BEEP, but I think it's pretty much dead and I wonder was it ever widely used.

Any ideas?

Answer 1

You might want to look at RTP (Real-time Transport Protocol), which might not be directly useful (and/or might be overkill), but its design would give you some good ideas to work from. And you might well be able to use it.

Answer 2

I think before getting into protocol design you should take care of the following issues really carefully:

• Almost all protocols except HTTP are filtered by firewalls these days. Even many content types and ports (except 80) may be filtered by service providers, especially in mobile data services. Therefore, make sure that you have no firewall problems in your target network before choosing to use or design a protocol.
• Size and speed matters. Try to use efficient protocols both in transport message size and encode/decode (serialize/deserialize) speed. Google Protocol Buffers provides a reliable solution to this problem.
• Connections always disconnect. You can never rely on a connection to remain open for long because of NAT timeouts (15 mins by default), protocol timeouts (30 mins for TCP) and many existing network problems. So, do not prefer a protocol over the other just because it keeps the connection open (it may try to, but never succeeds). Sending heartbeats is a good try for the timeout problem but network disconnects remain still unavoidable.
• Throughput matters. By throughput, I mean the number of messages processed in a period of time (e.g. 1 second). Using asynchronous protocols which disconnect a client after receiving its message, really helps in increasing the throughput. Although do not rely on connecting to the client from server to push the result because many clients are behind NATs and firewalls which avoid direct connection from outside. Try to keep the connection open or poll the server for the result. Avoiding state in a conversation is also the other method that helps scaling application throughput as the number of clients grow.
• There is no real-time in existing WANs. Do not trust those who say they have implemented a real-time protocol based on existing Wide Area Network protocols. You can never implement a real-time protocol on top of non-real-time ones. You can just do your best and pray for the network to go fast. That means: Do not stop, do your best.
• Non-Blocking IO. NIO (Non-blocking IO) is the trend now, effectively being used in network programming. It enables large number of connections with less memory usage and limited number of threads. Java 5 has native support for NIO which is very primitive. Many frameworks, such as Apache MINA and Netty, have been implemented based on Java NIO to make non-blocking programming easier and more robust. I strongly recommend them.

Answer 3

You might look at Kryo and/or KryoNet, which are NIO and Java-based and work the desktop and Android. You would have to write the iPhone side though, which would be rather tricky. Kryo beats Google's Protocol Buffers in serialized size (benchmarks here) and ease of use (no .proto type schema needs to be written, with Kryo the Java class definitions are the schema).

Regarding NIO, you might check out NPTL. The old becomes new again.

Answer 4

It's not an elegant solution, but you can do this over straight http, by not setting the content-length field on the http response, and never closing the output stream. Just keep sending data.

You could also set a very large content-length and send data back from the server in real time until the server has sent that amount, then the client can choose to make a new request or not.

Unfortunately i couldn't find any useful links for these ideas, but i trust you get the ideas.

The real plus side of these ideas is that they're over http, which gets by the firewall issue, and you can implement your app as a webapp with a servlet.

Just my 2 cents ;)