tags:

views:

1877

answers:

11

I'm looking to stream lots of data (up to ~1 Gbit) from Java to a C++ application (both on the same machine). I'm currently using a FIFO on Linux but need a Windows solution too.

The most cross-platform method seems to be a local socket, but: a) won't I get huge overhead from TCP checksumming and copying to & from kernel space, and b) won't the average user's firewall try to inspect the and maybe block the connection?

It seems like a safer solution may be to use JNI and the Named Pipe API (\.\pipe\blah), making a god-awful platform-specific mess of both sides of the connection.

Are these really my 2 best options (and which would people recommend?) Thanks!

+3  A: 

Named pipes would be more efficient than TCP, but how about just using shared memory blocks?

I don't know what primitives exist on the Java side for interfacing with shared memory, but from the C++ side it would be more efficient to access data in shared memory than read it out of either a socket or named pipe. You would have to implement your own flow control and blocking primitives, but these could be fairly straight forward.

Rob Walker
+4  A: 

I would use a local socket, which is, as you state, the most cross-platform method.

Kernel-User space copies should not be an issue since any other method you could choose would requiere this kind of copy except for maybe shared memory. It is available on every Unix system and also Windows has its way of doing it

To use shared memory in Java the only way is to implement it by means of your own .DLL/.SO and JNI to access it.

Fernando Miguélez
A: 

I would advise against JNI, because it is very difficult to debug. If the C++ code segfaults or throws an uncaught exception, your JVM will crash, and you will have no idea why.

Dima
This is not true. You can debug JNI code by attaching to the JVM.
JesperE
You are correct, if you are running on a PC. At my work, we are developing an embedded system running linux, which, unfortunately, does not have a native debugger. Not even gdb. So when the jni code crashes, it is very difficult to figure out why.
Dima
A: 

I would use a local socket with negative acknowledgment UDP if the bit rate is going to be too high for TCP (although I would try TCP first and confirm that it is a problem). There should be minimal, if any, packets dropped if you are doing the streaming on the same machine, but the addition of a negative acknowledgement layer will take care of that case for you.

Robin
A: 

How about using System.out and System.in?

If that isn't suitable, then Sockets is your best bet.

Pyrolistical
+4  A: 

You should take a look at Protocol Buffers from Google which supports both C++ and Java.

John Channing
A: 

If your C++ process starts the Java process, it might benefit from the inheritedChannel. Also, if the Java process is using a file, I recommend exploration of the transferTo and transferFrom methods. When doing file to file IO, these avoid needlessly tripping back and forth between user and kernel space; the same optimizations might kick in if you're using a special socket channel.

erickson
A: 

I recommend a UDP "connection" that acknowledges every Nth packet that was received without fault, and requests a retransmission of the few packets it will miss.

Ed
A: 

If you're happy with writing JNI, consider Boost.Interprocess. This will give you portable shared memory on both Linux and Windows. Remember that there's no kernel roundtrip for reading/writing shared memory.

Sunlight
A: 

Your fastest solution will be memory mapping a shared segment of memory, and them implementing a ring-buffer or other message passing mechanism. In C++ this is straight forward, and in Java you have the FileChannel.map method which makes it possible.

The next alternative would be to use the stdin/stdout of the two processes. If one can exec the other, this can be quite fast.

Lastly, as you've noted, you can do socket IO. For streaming video this isn't a great option, but if your passing XML around, the overhead will be minimal in comparison to the other processing.

brianegge
A: 

If it's a big chunk of data in "one" function call I would recommend JNI.

Take a look at this:Sharing output streams through a jni interface

Snippet from the article, it transfert data from c++ to java, the opposite would be also easy to do:

In all, the general strategy for sharing binary data (A/V files, images, etc.) from C with Java requires byte arrays. You create a Java byte array in C like this:

const char[] rawData = {0,1,2,3,4,5,6,7,8,9}; //Or get some raw data from somewhere
int dataSize = sizeof(rawData);
printf("Building raw data array copy\n");
jbyteArray rawDataCopy = env->NewByteArray(dataSize);
env->SetByteArrayRegion(rawDataCopy, 0, dataSize, rawData);

And pass it to Java like this:

printf("Finding callback method\n");
//Assumes obj is the Java instance that will receive the raw data via callback
jmethodID aMethodId = env->GetMethodID(env->GetObjectClass(obj),"handleData","([B)V");
if(0==aMethodId) throw MyRuntimeException("Method not found error");
printf("Invoking the callback\n");
env->CallVoidMethod(obj,aMethodId, &rawDataCopy);

you would have a Java object that looked something like this:

public class MyDataHandler {
  OutputStream dataStream;
  public MyDataHandler(OutputStream writeTo) { dataStream = writeTo;}
  public void handleData(byte[] incomingData) { dataStream.write(incomingData); }
}

That handler would be passed to C via native method like so:

public class NativeIntegration {
  public native void generateBinaryWithHandler(MyDataHandler handler);

  //Here we assume response is something like a network stream
  public void doCallNativeFunction(ResponseStream response) {
    MyDataHandler handler = new MyDataHandler(response);
    generateBinaryWithHandler(handler);
  }
}

Also, you can use other technologies: CORBA, ASN.1 (ASN.1 tool), UDP or TCP

Vincent